Image filtering, coding and decoding method and device based on neural network, equipment and storage medium

Abstract

The invention provides an image filtering, encoding and decoding method and device based on a neural network, equipment and a storage medium, the method can be applied to the fields of image processing, video encoding and decoding, video live broadcast and the like, and the method comprises the following steps: based on a to-be-filtered image, determining input information of a neural network filter, the input information comprising the to-be-filtered image; filtering the input information through a neural network filter to obtain a filtered image of the to-be-filtered image; wherein the neural network filter comprises a plurality of convolution layers used for extracting image feature information, and the plurality of convolution layers are obtained by processing a standard convolution layer through at least two modes of a multistage receptive field mode, a tensor decomposition mode and a grouping convolution mode. Therefore, under the condition that the filtering performance of the neural network filter is not reduced, the filtering operation complexity of the neural network filter is reduced, and the filtering effect of the image is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a neural network-based image filtering and encoding and decoding method, device, equipment, and storage medium. Background Art

[0002] With the development of video technology, the amount of data included in video data is large. In order to facilitate the transmission of video data, video devices implement video compression technology to make video data more efficiently transmitted or stored. In video compression, both the encoding end and the decoding end need to obtain reconstructed images through operations such as inverse quantization and inverse transformation. Since loss is introduced in video compression, the reconstructed image is filtered to reduce the compression loss of the image.

[0003] With the rapid development of neural network technology, neural network filters are widely used in video processing. However, the current neural network filters cannot strike a balance between filtering performance and filtering complexity, resulting in poor image filtering effects. Summary of the invention

[0004] The present application provides a neural network-based image filtering and encoding and decoding method, device, equipment, and storage medium, which reduces the complexity of filtering operations while ensuring image filtering performance, thereby improving the filtering effect.

[0005] In a first aspect, the present application provides an image filtering method based on a neural network, comprising:

[0006] Based on the image to be filtered, determining input information of the neural network filter, wherein the input information includes the image to be filtered;

[0007] The input information is filtered by a neural network filter to obtain a filtered image of the image to be filtered;

[0008] Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

[0009] In a second aspect, the present application provides an image decoding method, comprising:

[0010] Decoding a code stream, determining a reconstructed image of a current image, and determining input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image;

[0011] The input information is filtered by a neural network filter to obtain a filtered image of the reconstructed image;

[0012] Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

[0013] In a third aspect, the present application provides an image encoding method, comprising:

[0014] Encoding a current image, determining a reconstructed image of the current image, and determining input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image;

[0015] The input information is filtered by a neural network filter to obtain a filtered image of the reconstructed image;

[0016] Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

[0017] In a fourth aspect, the present application provides an image filtering device based on a neural network, comprising:

[0018] A determination unit, configured to determine input information of a neural network filter based on the image to be filtered, wherein the input information includes the image to be filtered;

[0019] A filtering unit, filtering the input information through a neural network filter to obtain a filtered image of the image to be filtered;

[0020] Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

[0021] In a fifth aspect, the present application provides an image decoding device, comprising:

[0022] A decoding unit, configured to decode a code stream, determine a reconstructed image of a current image, and determine input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image;

[0023] A filtering unit, configured to filter the input information through a neural network filter to obtain a filtered image of the reconstructed image;

[0024] Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

[0025] In a sixth aspect, the present application provides an image encoding device, including:

[0026] An encoding unit, configured to encode a current image, determine a reconstructed image of the current image, and determine input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image;

[0027] A filtering unit, configured to filter the input information through a neural network filter to obtain a filtered image of the reconstructed image;

[0028] Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

[0029] In a seventh aspect, a decoder is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the second aspect or its implementations.

[0030] In an eighth aspect, an encoder is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the third aspect or its implementations.

[0031] In a ninth aspect, a chip is provided for implementing the method in any one of the first to third aspects or their respective implementations. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the method in any one of the first to third aspects or their respective implementations.

[0032] In a tenth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method in any one of the first to third aspects or any of its implementations.

[0033] In an eleventh aspect, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions enable a computer to execute the method in any one of the first to third aspects or any of their implementations.

[0034] In a twelfth aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method in any one of the first to third aspects or its respective implementation manners.

[0035] In summary, the present application proposes a new neural network filter, which includes multiple convolutional layers for extracting image feature information, and these multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways among a multi-level receptive field method, a tensor decomposition method, and a grouped convolution method. That is, the present application decomposes the existing standardized convolution operation into a convolution operation with low computational complexity through at least two ways among a multi-level receptive field method, a tensor decomposition method, and a grouped convolution method, thereby reducing the filtering operation complexity of the neural network filter without reducing the filtering performance of the neural network filter in advance, thereby improving the filtering effect of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 A schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of a coding framework provided in an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of a decoding framework provided in an embodiment of the present application;

[0040] Figure 4 is a schematic diagram of a coding unit;

[0041] Figure 5 A schematic diagram of the filtering process of a neural network filter;

[0042] Figure 6 A schematic diagram of a single-branch neural network filter;

[0043] FIG. 7A to FIG. 7C Schematic diagram of an existing residual block;

[0044] Figure 8 A schematic diagram of a dual-branch neural network filter;

[0045] Fig. 9 A flowchart of a neural network-based image filtering method provided in one embodiment of the present application;

[0046] Fig.10 It is a schematic diagram of CP decomposition;

[0047] Fig.11A , Fig. 11B , Fig. 12A and Fig. 12B Schematic diagrams of several networks after processing the head or tail convolutional layer in the neural network filter;

[0048] Fig.13 A schematic diagram of a neural network filter involved in an embodiment of the present application;

[0049] Fig.14 A schematic diagram of another neural network filter involved in an embodiment of the present application;

[0050] FIG. 15A to FIG. 15E Some network structure diagrams of the i-th residual block;

[0051] FIG. 16A to FIG. 19D Some network structure diagrams of the i-th residual block;

[0052] Fig. 20 is a structural diagram of the i-th residual block;

[0053] FIG. 21A to FIG. 21D Some network structure diagrams of the i-th residual block;

[0054] FIG. 22A to FIG. 23C Some network structure diagrams of neural network filters;

[0055] Fig.24 A schematic diagram of a flow chart of an image decoding method provided in an embodiment of the present application;

[0056] Fig.25 A schematic diagram of a flow chart of an image filtering method provided in one embodiment of the present application;

[0057] Fig.26 is a schematic block diagram of an image filtering device based on a neural network provided in one embodiment of the present application;

[0058] Fig. 27 is a schematic block diagram of an image decoding device provided by an embodiment of the present application;

[0059] Fig.28 is a schematic block diagram of an image encoding device provided by an embodiment of the present application;

[0060] Fig.29 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0062] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In an embodiment of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units does not have to be limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, unless otherwise specified, "multiple" refers to two or more than two.

[0063] The present application can be applied to the field of image coding and decoding, the field of video coding and decoding, the field of hardware video coding and decoding, the field of dedicated circuit video coding and decoding, the field of real-time video coding and decoding, etc. For example, the scheme of the present application can be combined with the standard of end-to-end image coding based on deep learning, such as JPEG AI. Alternatively, the scheme of the present application can be combined with other exclusive or industry standards and operate, and the standards include ITU-TH.261, ISO / IECMPEG-1Visual, ITU-TH.262 or ISO / IECMPEG-2Visual, ITU-TH.263, ISO / IECMPEG-4Visual, ITU-TH.264 (also known as ISO / IECMPEG-4AVC), including scalable video coding (SVC) and multi-view video coding (MVC) extensions. It should be understood that the technology of the present application is not limited to any specific coding standard or technology.

[0064] To facilitate understanding, first combine Figure 1 The video encoding and decoding system involved in the embodiments of the present application is introduced.

[0065] Figure 1This is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application. It should be noted that: Figure 1 This is just an example. The video encoding and decoding system of the embodiment of the present application includes but is not limited to Figure 1 As shown. Figure 1 As shown, the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device is used to encode (which can be understood as compressing) video data to generate a code stream, and transmit the code stream to the decoding device. The decoding device decodes the code stream generated by the encoding device to obtain decoded video data.

[0066] The encoding device 110 of the embodiment of the present application can be understood as a device with a video encoding function, and the decoding device 120 can be understood as a device with a video decoding function, that is, the embodiment of the present application includes a wider range of devices for the encoding device 110 and the decoding device 120, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, vehicle-mounted computers, etc.

[0067] In some embodiments, the encoding device 110 may transmit the encoded video data (eg, a code stream) to the decoding device 120 via the channel 130. The channel 130 may include one or more media and / or devices capable of transmitting the encoded video data from the encoding device 110 to the decoding device 120.

[0068] In one example, the channel 130 includes one or more communication media that enable the encoding device 110 to transmit the encoded video data directly to the decoding device 120 in real time. In this example, the encoding device 110 can modulate the encoded video data according to the communication standard and transmit the modulated video data to the decoding device 120. The communication medium includes a wireless communication medium, such as a radio frequency spectrum, and optionally, the communication medium may also include a wired communication medium, such as one or more physical transmission lines.

[0069] In another example, the channel 130 includes a storage medium, which can store the video data encoded by the encoding device 110. The storage medium includes a variety of locally accessible data storage media, such as optical disks, DVDs, flash memories, etc. In this example, the decoding device 120 can obtain the encoded video data from the storage medium.

[0070] In another example, the channel 130 may include a storage server that can store the video data encoded by the encoding device 110. In this example, the decoding device 120 can download the stored encoded video data from the storage server. Alternatively, the storage server can store the encoded video data and transmit the encoded video data to the decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0071] In some embodiments, the encoding device 110 includes a video encoder 112 and an output interface 113. The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0072] In some embodiments, the encoding device 110 may further include a video source 111 in addition to the video encoder 112 and the input interface 113 .

[0073] The video source 111 may include at least one of a video acquisition device (eg, a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data.

[0074] The video encoder 112 encodes the video data from the video source 111 to generate a code stream. The video data may include one or more pictures or a sequence of pictures. The code stream contains the coding information of the picture or the sequence of pictures in the form of a bit stream. The coding information may include the coded picture data and associated data. The associated data may include a sequence parameter set (SPS for short), a picture parameter set (PPS for short) and other syntax structures. The SPS may contain parameters applied to one or more sequences. The PPS may contain parameters applied to one or more pictures. The syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the code stream.

[0075] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored in a storage medium or a storage server for subsequent reading by the decoding device 120.

[0076] In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122 .

[0077] In some embodiments, the decoding device 120 may include a display device 123 in addition to the input interface 121 and the video decoder 122 .

[0078] The input interface 121 includes a receiver and / or a modem. The input interface 121 can receive the encoded video data through the channel 130 .

[0079] The video decoder 122 is used to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to the display device 123 .

[0080] The display device 123 displays the decoded video data. The display device 123 may be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 may include a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0081] also, Figure 1 This is only an example, and the technical solution of the embodiment of the present application is not limited to Figure 1 For example, the technology of the present application can also be applied to single-sided video encoding or single-sided video decoding.

[0082] The following is an introduction to the video encoding framework involved in the embodiments of the present application.

[0083] Figure 2 2 is a schematic block diagram of a video encoder involved in an embodiment of the present application. It should be understood that the video encoder 200 can be used to perform lossy compression on an image, or can be used to perform lossless compression on an image. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0084] The video encoder 200 can be applied to image data in luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2 or 4:4:4, Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 means full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0085] For example, the video encoder 200 reads video data, and for each frame of the video data, divides the frame into a number of coding tree units (CTUs). In some examples, CTUs may be referred to as "tree blocks", "largest coding units" (LCU) or "coding tree blocks" (CTB). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to a luminance (luminance or luma) sample and two chrominance (chrominance or chroma) samples. Therefore, each CTU may be associated with a luminance sample block and two chrominance sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding, and a CU may be a rectangular block or a square block. CU can be further divided into prediction unit (PU) and transform unit (TU), so that coding, prediction and transform are separated and more flexible in processing. In one example, CTU is divided into CU in quadtree mode, and CU is divided into TU and PU in quadtree mode.

[0086] The video encoder and video decoder may support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder may support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N or similar sizes for inter-frame prediction. The video encoder and video decoder may also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0087] In some embodiments, Figure 2 As shown, the video encoder 200 may include: a prediction unit 210, a convolution unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filter unit 260, a decoded image buffer 270 and an entropy coding unit 280. It should be noted that the video encoder 200 may include more, fewer or different functional components.

[0088] Optionally, in the present application, the current block may be referred to as a current coding unit (CU) or a current prediction unit (PU), etc. A prediction block may also be referred to as a prediction image block or an image prediction block, and a reconstructed image block may also be referred to as a reconstructed block or an image reconstructed image block.

[0089] In some embodiments, the prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame prediction unit 212. Since there is a strong correlation between adjacent pixels in a frame of a video, the intra-frame prediction method is used in the video coding and decoding technology to eliminate the spatial redundancy between adjacent pixels. Since there is a strong similarity between adjacent frames in a video, the inter-frame prediction method is used in the video coding and decoding technology to eliminate the temporal redundancy between adjacent frames, thereby improving the coding efficiency.

[0090] The inter-frame prediction unit 211 can be used for inter-frame prediction, which may include motion estimation and motion compensation. Motion estimation can search the reference pictures in the list of reference pictures to find the reference block of the image block to be encoded. Motion estimation can generate an index indicating the reference block and a motion vector indicating the spatial displacement between the image block to be encoded and the reference block. Motion estimation can output the index of the reference block and the motion vector as the motion information of the image block to be encoded. Motion compensation can obtain the prediction information of the image block to be encoded based on the motion information of the image block to be encoded. Inter-frame prediction can refer to the image information of different frames. Inter-frame prediction uses motion information to find the reference block from the reference frame, and generates a prediction block based on the reference block to eliminate temporal redundancy; the frame used for inter-frame prediction can be a P frame and / or a B frame, P frame refers to a forward prediction frame, and B frame refers to a bidirectional prediction frame. Inter-frame prediction uses motion information to find the reference block from the reference frame, and generates a prediction block based on the reference block. The motion information includes the reference frame list where the reference frame is located, the reference frame index, and the motion vector. The motion vector can be an integer pixel or a sub-pixel. If the motion vector is a sub-pixel, then it is necessary to use interpolation filtering in the reference frame to make the required sub-pixel block. Here, the integer pixel or sub-pixel block in the reference frame found according to the motion vector is called a reference block. Some technologies will directly use the reference block as a prediction block, while some technologies will generate a prediction block based on the reference block. Generating a prediction block based on the reference block can also be understood as using the reference block as a prediction block and then processing it based on the prediction block to generate a new prediction block.

[0091] The intra-frame prediction unit 212 only refers to the information of the same frame image to predict the pixel information in the current coded image block to eliminate spatial redundancy. The frame used for intra-frame prediction may be an I frame.

[0092] There are multiple prediction modes for intra-frame prediction. Taking the H series of international digital video coding standards as an example, the H.264 / AVC standard has 8 angle prediction modes and 1 non-angle prediction mode, and H.265 / HEVC is extended to 33 angle prediction modes and 2 non-angle prediction modes. The intra-frame prediction modes used by HEVC are Planar, DC, and 33 angle modes, for a total of 35 prediction modes. The intra-frame modes used by VVC are Planar, DC, and 65 angle modes, for a total of 67 prediction modes.

[0093] It should be noted that with the increase of angle modes, intra-frame prediction will be more accurate and more in line with the needs of the development of high-definition and ultra-high-definition digital videos.

[0094] Convolution unit 220 may generate a residual block of a CU based on the pixel block of the CU and the prediction block of the PU of the CU. For example, convolution unit 220 may generate a residual block of the CU so that each sample in the residual block has a value equal to the difference between the following two: a sample in the pixel block of the CU and a corresponding sample in the prediction block of the PU of the CU.

[0095] The transform / quantization unit 230 can quantize the transform coefficients. The transform / quantization unit 230 can quantize the transform coefficients associated with the TU of the CU based on the quantization parameter (QP) value associated with the CU. The video encoder 200 can adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU. Exemplarily, the residual video signal is converted into a transform domain through transformation operations such as DFT and DCT, which are called transform coefficients. The signal in the transform domain is further subjected to a lossy quantization operation, which loses certain information, making the quantized signal conducive to compressed expression. In some video coding standards, there may be more than one transform method to choose from, so the encoder also needs to select one of the transforms for the current encoded CU and inform the decoder. The degree of quantization fineness is usually determined by the quantization parameter (QP). A larger QP value means that coefficients with a larger value range will be quantized to the same output, which usually results in greater distortion and a lower bit rate. Conversely, a smaller QP value means that coefficients with a smaller value range will be quantized to the same output, which usually results in less distortion and a corresponding higher bit rate.

[0096] The inverse transform / quantization unit 240 may apply inverse quantization and inverse transform to the quantized transform coefficients, respectively, to reconstruct a residual block from the quantized transform coefficients.

[0097] The reconstruction unit 250 may add the samples of the reconstructed residual block to the corresponding samples of one or more prediction blocks generated by the prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of the CU in this manner, the video encoder 200 may reconstruct the pixel blocks of the CU.

[0098] The loop filter unit 260 is used to process the pixels after inverse transformation and inverse quantization, compensate for the distortion information, and provide a better reference for the subsequent encoded pixels. For example, a deblocking filter operation can be performed to reduce the blocking effect of the pixel blocks associated with the CU. As can be seen from the above, the encoded image can be reconstructed and decoded after inverse quantization, inverse transformation and prediction compensation. Compared with the original image, the reconstructed image has some information different from the original image due to the influence of quantization, resulting in distortion. Filtering the reconstructed image, such as deblocking filter DBF (deblocking filter), SAO (sample adaptive offset) or ALF (adaptive loop filter) and other filters, can effectively reduce the degree of distortion caused by quantization. Since these filtered reconstructed images will be used as references for subsequent encoded images and used to predict future signals, the above filtering operation is also called loop filtering, and filtering operations within the encoding loop.

[0099] The decoded image buffer 270 may store the reconstructed pixel blocks. The inter prediction unit 211 may use the reference image containing the reconstructed pixel blocks to perform inter prediction on PUs of other images. In addition, the intra prediction unit 212 may use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra prediction on other PUs in the same image as the CU.

[0100] The entropy coding unit 280 may receive the quantized transform coefficients from the transform / quantization unit 230. The entropy coding unit 280 may perform one or more entropy coding operations on the quantized transform coefficients to generate entropy coded data. Exemplarily, the quantized transform domain signal will be statistically compressed and encoded according to the frequency of occurrence of each value, and finally a binary (0 or 1) compressed code stream will be output. At the same time, the encoding generates other information, such as the selected mode, motion vector, etc., which also needs to be entropy encoded to reduce the bit rate. In one example, statistical coding can be used as a lossless coding method that can effectively reduce the bit rate required to express the same signal. Common statistical coding methods include variable length coding (VLC) or context-based binary arithmetic coding (CABAC, Content Adaptive Binary Arithmetic Coding).

[0101] Figure 3 It is a schematic block diagram of a video decoder involved in an embodiment of the present application.

[0102] like Figure 3As shown, the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transformation unit 330, a reconstruction unit 340, a loop filter unit 350, and a decoded image buffer 360. It should be noted that the video decoder 300 may include more, fewer, or different functional components.

[0103] The video decoder 300 may receive a bitstream. The entropy decoding unit 310 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding unit 310 may parse the syntax elements in the bitstream that have been entropy encoded. The prediction unit 320, the inverse quantization / transformation unit 330, the reconstruction unit 340, and the loop filter unit 350 may decode the video data according to the syntax elements extracted from the bitstream, that is, generate decoded video data.

[0104] In some embodiments, the prediction unit 320 includes an intra prediction unit 322 and an inter prediction unit 321 .

[0105] The intra prediction unit 322 may perform intra prediction to generate a prediction block for the PU. The intra prediction unit 322 may use an intra prediction mode to generate a prediction block for the PU based on pixel blocks of spatially neighboring PUs. The intra prediction unit 322 may also determine the intra prediction mode of the PU according to one or more syntax elements parsed from the code stream.

[0106] The inter prediction unit 321 may construct a first reference image list (list 0) and a second reference image list (list 1) according to the syntax elements parsed from the code stream. In addition, if the PU is encoded using inter prediction, the entropy decoding unit 310 may parse the motion information of the PU. The inter prediction unit 321 may determine one or more reference blocks of the PU according to the motion information of the PU. The inter prediction unit 321 may generate a prediction block of the PU according to one or more reference blocks of the PU.

[0107] The inverse quantization / transform unit 330 may inversely quantize (ie, dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 330 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0108] After inverse quantizing the transform coefficients, the inverse quantization / transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.

[0109] The reconstruction unit 340 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 may add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0110] The loop filtering unit 350 may perform a deblocking filtering operation to reduce blocking effects of pixel blocks associated with a CU.

[0111] The video decoder 300 may store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 may use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or input the reconstructed image into a display device for display.

[0112] The basic process of video encoding and decoding is as follows: at the encoding end, a frame of image is divided into blocks, and for the current block, the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block of the current block. The convolution unit 220 can calculate the residual block based on the original block of the prediction block and the current block, that is, the difference between the original block of the prediction block and the current block, and the residual block can also be called residual information. The residual block can remove information that is not sensitive to the human eye through the transformation and quantization process of the transformation / quantization unit 230 to eliminate visual redundancy. Optionally, the residual block before transformation and quantization by the transformation / quantization unit 230 can be called a time domain residual block, and the time domain residual block after transformation and quantization by the transformation / quantization unit 230 can be called a frequency residual block or a frequency domain residual block. The entropy coding unit 280 receives the quantized change coefficient output by the change quantization unit 230, and can entropy encode the quantized change coefficient and output a bit stream. For example, the entropy coding unit 280 can eliminate character redundancy according to the target context model and the probability information of the binary bit stream.

[0113] At the decoding end, the entropy decoding unit 310 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction to generate a prediction block of the current block based on the prediction information. The inverse quantization / transformation unit 330 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or on the block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be used as a reference frame for inter-frame prediction for subsequent frames.

[0114] It should be noted that the block division information determined by the encoder, as well as the mode information or parameter information such as prediction, transformation, quantization, entropy coding, loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder by analyzing the existing information, thereby ensuring that the decoded image obtained by the encoder is the same as the decoded image obtained by the decoder.

[0115] The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. The present application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0116] In the existing hybrid coding framework, each frame of a video is often divided into units of a certain size before the subsequent encoding and decoding process. Figure 4 As shown in the figure, the largest coding unit (CTU) is the basic coding unit in the hybrid coding framework, which usually contains two parts: luminance Y and chrominance UV. Since the U component and V component in chrominance have similar characteristics, they are usually processed in the order of U first and then V using the same coding parameters, and the corresponding coding results of U and V are obtained.

[0117] The existing hybrid coding framework uses traditional loop filters to suppress the distortion of reconstructed images, improve the quality of reconstructed images, and expects to restore the encoded reconstructed images to the original images. However, traditional loop filters are based on manual design and are difficult to effectively reduce the distortion of reconstructed images, leaving a large room for optimization. Due to the excellent performance of deep learning tools in image processing, deep learning-based loop filters are applied to the loop filter module.

[0118] The main technology involved in this application is a filter based on a neural network (Neural Network Loop Filter, NNLF). Figure 5 As shown, by inputting the image to be filtered before filtering into the trained filter, the enhanced image after filtering can be obtained.

[0119] During the training process, neural networks usually use loss functions to constrain the filtered image so that it can be restored to the original image as much as possible. The loss function measures the difference between the filtered value and the true value. The larger the loss value, the greater the difference, and the goal of training is to reduce the loss. For deep learning-based encoding tools, exemplary and commonly used loss functions are: L1 norm loss function, L2 norm loss function and smooth L1 loss function.

[0120] Currently, commonly used neural network filters include convolutional neural network (CNN) filters. Convolution is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks are mainly composed of a series of basic convolutional layers or convolutional modules built based on convolutional layers. Figure 6It is a specific example of a network structure model. The network includes a head, a body and a tail, wherein the body is composed of at least one residual block (ResidualBlock, ResBlock / RB).

[0121] ResBlock is a classic convolutional module that is used in various neural network models. Its common structure is as follows: FIG. 7A to FIG. 7C shown.

[0122] The specific usage of ResBlock is as follows:

[0123] Mode 1, input is image-level information: The input of ResBlock can be one or more of image information such as reconstructed image (rec), predicted image (rec) and edge information such as boundary strength (BS), frame quantization parameter (sliceQP), base quantization parameter (baseQP), frame type (sliceType / IPB). The output is a feature-level image with K channels (K is a positive integer, when K=1, the output is image-level information).

[0124] Mode 2, input is feature-level information: The input of ResBlock can be the feature information with K channels extracted by the previous convolutional layer / convolutional block. At this time, the output of ResBlock is the feature-level information with M channels (K and M are both positive integers. When M=1, the output is image-level information).

[0125] In addition, different neural network models can use ResBlock to replace any convolutional layer or convolutional block in the model. Figure 6 In different network structures, the network separates the brightness component and the chrominance component, but ResBlock can still be used in the structure.

[0126] like Figure 6 or Figure 8 As shown in the figure, existing neural network filters often build the overall network structure based on commonly used basic modules such as 3x3 convolution and residual blocks to train neural network filters. For the network structure of the neural network filter, repeated stacking of 3x3 convolution and residual blocks can bring considerable filtering performance, but it introduces too high computational complexity. However, limiting the number of repetitions of 3x3 convolution and residual blocks, that is, specifying a certain computational complexity, makes it difficult to obtain higher filtering performance. Therefore, how to obtain higher image filtering performance without introducing too much computational complexity is an important research direction of image filtering technology based on neural networks.

[0127] Based on this, in order to solve the technical problem, the embodiment of the present application proposes a new neural network filter, which includes multiple convolutional layers for extracting image feature information. The multiple convolutional layers are obtained by processing the standard convolutional layers in at least two ways among the multi-level receptive field method, the tensor decomposition method and the grouped convolution method. For example, the embodiment of the present application replaces the standard convolutional layer in the existing neural network filter, such as Figure 6 or Figure 8 The standard convolutional layers (e.g., 3x3 convolutional layers) in the head, main body, tail, etc. of the neural network filter shown are processed by at least two of the multi-level receptive field method, the tensor decomposition method, and the grouped convolution method, and converted into convolution operations with low computational complexity, thereby reducing the filtering operation complexity of the neural network filter without reducing the filtering performance of the neural network filter in advance, thereby improving the filtering effect of the image.

[0128] The technical solutions of the embodiments of the present application are described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0129] First, taking the decoding end as an example, the image decoding method provided in the embodiment of the present application is introduced.

[0130] Fig. 9 A flowchart of a neural network-based image filtering method provided in an embodiment of the present application. The present application embodiment is applied to Figure 1 , Figure 2 or Figure 3 The decoder or decoding device, and the encoder or encoding device shown can also be applied to other electronic devices. For ease of description, the following is an example of an execution subject being a filtering device. The filtering device can be understood as any electronic device with an image filtering function, such as the above-mentioned decoding device or encoding device.

[0131] like Fig. 9 As shown, the method of the embodiment of the present application includes:

[0132] S101. Determine input information of a neural network filter based on an image to be filtered.

[0133] The above-mentioned input information includes the image to be filtered.

[0134] The image to be filtered in the embodiment of the present application can be understood as any image that needs to be filtered, such as a reconstructed image or any other image to be filtered.

[0135] In an example, the image to be filtered may be an image, such as a frame of reconstructed image or a whole reconstructed image.

[0136] In an example, the image to be filtered may be a local area in an image, for example, a part of a reconstructed area of ​​a reconstructed image.

[0137] The embodiment of the present application does not impose any specific restrictions on the specific content of the input information of the neural network filter, as long as the input information includes the image to be filtered.

[0138] For example, in some embodiments, the input information of the embodiments of the present application only includes the image to be filtered and does not include other information.

[0139] In some embodiments, the input information of the embodiments of the present application includes, in addition to the image to be filtered, other related content of the image to be filtered. For example, if the image to be filtered is a reconstructed image, the input information includes, in addition to the reconstructed image, at least one of the predicted image corresponding to the reconstructed image, the boundary strength, the frame type (slicetype) of the reconstructed image, and the quantization parameter information. Optionally, the quantization parameter information includes sliceQP (slice level quantization parameter) and base QP (basic quantization parameter).

[0140] In some embodiments, the input information may also include other related information of the image to be filtered, which is not limited in this embodiment of the present application.

[0141] Based on the above steps, the filtering device determines the input information of the neural network filter and then executes the following step S102.

[0142] S102, filtering the input information through a neural network filter to obtain a filtered image of the image to be filtered.

[0143] The neural network filter includes a plurality of convolutional layers for extracting image feature information, and the plurality of convolutional layers are obtained by processing the standard convolutional layers in at least two ways including a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

[0144] The existing neural network filter can bring considerable filtering performance through repeated stacking of 3x3 convolution and residual blocks, but it introduces too high computational complexity and cannot achieve a balance between filtering performance and computational complexity. In order to solve this technical problem, the embodiment of the present application proposes a new neural network filter, which includes multiple convolution layers for extracting image feature information. These multiple convolution layers are obtained by processing standard convolution layers in at least two of the multi-level receptive field method, tensor decomposition method and grouped convolution method. For example, the embodiment of the present application replaces the standard convolution layer in the existing neural network filter, such as Figure 6 or Figure 8At least one standard convolution layer (for example, a 3x3 convolution layer) in the head, main body, tail, etc. of the neural network filter shown is processed by at least two of the multi-level receptive field method, the tensor decomposition method, and the grouped convolution method, and converted into a convolution operation with low computational complexity, thereby reducing the filtering operation complexity of the neural network filter without reducing the filtering performance of the neural network filter in advance, thereby improving the filtering effect of the image.

[0145] The following introduces related technologies such as multi-level receptive field, tensor decomposition and group convolution.

[0146] Multi-Level Receptive Field (MLRF), in a convolutional neural network, the size of the area in which a single element of the output feature map of each network layer is mapped to the input feature map is called the receptive field. Existing NNLFs usually use ordinary 1x1 convolutional layers or 3x3 convolutional layers as the basic components of the network structure. Due to the different sizes of the convolutional windows, the number of elements of the processed input feature maps is different, and different convolutional layers usually have different receptive field sizes. In order to effectively extract and fuse the input feature information, the parallel use of 1x1 convolutional layers and 3x3 convolutional layers can obtain feature information at different receptive field levels, thereby improving the generalization of the neural network model.

[0147] Canonical Polyadic Decomposition (CPD), also known as CP decomposition, is a classic tensor decomposition method that can be used in neural networks to reduce the amount of convolution calculations. Fig.10 As shown in the figure, according to the principle of CP decomposition, a 3x3 convolution can be approximated by a 1x3 convolution and a 3x1 convolution, thereby effectively reducing the complexity of the convolution operation without significantly reducing the network performance.

[0148] Group Convolution (GC) is a variant of standard convolution. It divides the number of channels of the input convolution into G groups, and then performs standard convolution on each group of features. Since the number of input channels of each group becomes 1 / G of the original, the number of channels of the corresponding convolution kernel also becomes 1 / G of the original. In this way, group convolution can reduce the number of parameters and operations to 1 / G of the original, thereby effectively reducing the complexity of convolution operations.

[0149] The neural network filter of the embodiment of the present application can be understood as processing at least one standard convolution layer in the existing neural network filter by at least two of the multi-level receptive field method, the tensor decomposition method and the grouped convolution method to obtain any one neural network filter of multiple convolution layers with low computational complexity.

[0150] In some embodiments, the present application embodiments may be Figure 6 At least one standard convolutional layer at the head of the neural network filter shown is processed by at least two of the multi-level receptive field method, the tensor decomposition method and the grouped convolution method, and converted into multiple convolutional layers with lower complexity.

[0151] In one example, Figure 6 The 3X3 convolution layer in the head of the neural network filter shown in FIG. 1 is decomposed into a 1X3 convolution layer and a 3X1 convolution layer by tensor decomposition. Fig.11A As shown, the input feature information of the 1X3 convolution layer is divided into multiple groups through group convolution, and each group of feature information is convolved separately and then the channels are merged to serve as the output information of the 1X3 convolution layer. Fig.11A As shown in the figure, the input feature information of the 3X1 convolution layer can be divided into multiple groups by group convolution, and each group of feature information can be convolved separately and then the channels can be merged to serve as the output information of the 3X1 convolution layer.

[0152] In one example, Figure 6 The 3X3 convolution layer in the head of the neural network filter shown in FIG. 1 is converted into two parallel-connected 1X1 convolution layers and 3X3 convolution layers through a multi-level receptive field method. Then, the 3X3 convolution layer in the parallel branch is decomposed into a 1X3 convolution layer and a 3X1 convolution layer through a tensor decomposition method. Then, as Fig. 11B As shown, the input feature information of the 1X3 convolution layer is divided into multiple groups through group convolution, and each group of feature information is convolved separately and then the channels are merged to serve as the output information of the 1X3 convolution layer. Fig. 11B As shown in the figure, the input feature information of the 3X1 convolution layer can be divided into multiple groups by group convolution, and each group of feature information can be convolved separately and then the channels can be merged to serve as the output information of the 3X1 convolution layer.

[0153] In some embodiments, the present application embodiments may be Figure 6 At least one standard convolutional layer at the tail of the neural network filter shown is processed by at least two of a multi-level receptive field method, a tensor decomposition method, and a grouped convolution method, and converted into a plurality of convolutional layers with lower complexity.

[0154] In one example, Figure 6 The 3X3 convolution layer at the tail of the neural network filter shown in FIG. 1 is decomposed into a 1X3 convolution layer and a 3X1 convolution layer by tensor decomposition. Fig. 12AAs shown, the input feature information of the 1X3 convolution layer is divided into multiple groups through group convolution, and each group of feature information is convolved separately and then the channels are merged to serve as the output information of the 1X3 convolution layer. Fig. 12A As shown in the figure, the input feature information of the 3X1 convolution layer can be divided into multiple groups by group convolution, and each group of feature information can be convolved separately and then the channels can be merged to serve as the output information of the 3X1 convolution layer.

[0155] In one example, Figure 6 The 3X3 convolution layer in the tail of the neural network filter shown in the figure is converted into two parallel connected 1X1 convolution layers and 3X3 convolution layers through the multi-level receptive field method. Then, the 3X3 convolution layer in the parallel branch is decomposed into 1X3 convolution layers and 3X1 convolution layers through tensor decomposition. Then, as Fig. 12B As shown, the input feature information of the 1X3 convolution layer is divided into multiple groups through group convolution, and each group of feature information is convolved separately and then the channels are merged to serve as the output information of the 1X3 convolution layer. Fig. 12B As shown in the figure, the input feature information of the 3X1 convolution layer can be divided into multiple groups by group convolution, and each group of feature information can be convolved separately and then the channels can be merged to serve as the output information of the 3X1 convolution layer.

[0156] In some embodiments, the neural network filter of the embodiment of the present application includes a residual unit, which includes a plurality of residual blocks connected in sequence, and at least one residual block among the plurality of residual blocks, for example, the i-th residual block includes all or part of the convolution layers among the plurality of convolution layers obtained by processing the standard convolution layer by at least two of the multi-level receptive field method, the tensor decomposition method, and the grouped convolution method, and i is a positive integer. That is to say, the embodiment of the present application proposes a new residual block structure, which can be understood as processing at least one standard convolution layer in the existing residual block by at least two of the multi-level receptive field method, the tensor decomposition method, and the grouped convolution method to obtain a plurality of convolution layers with lower computational complexity.

[0157] The residual unit in the embodiment of the present application can be understood as the above Figure 6 The main part of .

[0158] The neural network filter shown in this embodiment includes multiple residual blocks, at least one of which can be understood as FIG. 7A to FIG. 7C It is obtained by processing the standard convolutional layer in the existing residual block shown in . Figure 6The standard convolution layer of at least one residual block among the multiple residual blocks included in the main part of the neural network filter shown is processed in at least two of the multi-level receptive field method, the tensor decomposition method and the group convolution method to obtain at least one residual block of the embodiment of the present application. For the sake of ease of description, the i-th residual block is taken as an example for explanation.

[0159] For the ith residual block included in the residual unit of the neural network filter, the network structure of the ith residual block mainly includes the following two cases:

[0160] In case 1, the standard convolutional layer is processed based on the multi-level receptive field method and the grouped convolution method to obtain the i-th residual block.

[0161] Specifically, Fig.13 As shown, the i-th residual block of the embodiment of the present application includes a first convolution unit, a second convolution unit, and a third convolution unit, wherein the first convolution unit and the second convolution unit are connected in parallel and then connected in series with the third convolution unit. The above-mentioned first convolution unit includes a first convolution layer, the second convolution unit includes a second convolution layer, and the third convolution unit includes a third convolution layer. The convolution kernel sizes of the first convolution layer and the second convolution layer are different, and at least one of the first convolution layer, the second convolution layer and the third convolution layer includes a grouped convolution layer, which is used to divide the input feature information of the grouped convolution layer into multiple groups based on the number of channels through grouped convolution, and convolve each group of feature information separately and then merge the number of channels as the output information of the grouped convolution layer.

[0162] The embodiment of the present application does not limit the number of convolutional layers included in the first convolutional unit, the second convolutional unit, and the third convolutional unit. For example, at least one convolutional unit among the first convolutional unit, the second convolutional unit, and the third convolutional unit includes one or more convolutional layers.

[0163] In some embodiments, at least one of the first convolution unit, the second convolution unit, and the third convolution unit may include an activation function, etc. in addition to the convolution layer.

[0164] The activation function of the embodiment of the present application can be Sigmoid, ReLU, LeakyReLU, PReLU, ELU and other activation functions.

[0165] In one example, if Fig.14 As shown, the first convolution unit includes a first convolution layer and a first activation function, the second convolution unit includes a second convolution layer and a second activation function, and the third convolution unit includes a third convolution layer.

[0166] By the above Fig.13 and Fig.14It can be seen that the first convolutional layer and the second convolutional layer are connected in parallel, and the convolution kernel sizes of the first convolutional layer and the second convolutional layer are different. In this way, when the first convolutional layer and the second convolutional layer in parallel process the feature information, feature information at different receptive field levels can be obtained, thereby improving the generalization of the neural network filter.

[0167] For example, the input feature information of the i-th residual block is recorded as the i-1th feature information, the convolution kernel size of the first convolution layer is assumed to be 3X3, and the convolution kernel size of the second convolution layer is 1X1. Fig.13 or Fig.14 In the i-th residual block shown, the first convolution layer in the first convolution unit extracts the i-1th feature information on the 3X3 convolution window, and can obtain the feature information at the first receptive field level. At the same time, the second convolution layer in the second convolution unit extracts the i-1th feature information on the 1X1 convolution window, and can obtain the feature information at the second receptive field level, so that the feature information at different receptive field levels can be obtained. Then, after the feature information at different receptive field levels is extracted, it is input into the third convolution unit for feature extraction again, and then fused with the i-1th feature information to obtain the output feature information of the i-th residual block.

[0168] That is to say, the residual block in the embodiment of the present application adopts a multi-level receptive field method, which can obtain feature information at different receptive field levels, thereby improving the feature processing effect of the residual block.

[0169] In addition, if Fig.13 or Fig.14 As shown, at least one of the first convolutional layer, the second convolutional layer and the third convolutional layer includes a grouped convolutional layer, which is used to divide the input feature information of the grouped convolutional layer into multiple groups based on the number of channels through grouped convolution, and convolve each group of feature information separately and then merge the channels as the output information of the grouped convolutional layer.

[0170] In one example, if Fig.15AAs shown, the first convolution layer includes a grouped convolution layer, while the second convolution layer and the third convolution layer are not grouped convolution layers. Assume that the input information of the first convolution layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this way, the filtering device inputs the i-1th feature information into the i-th residual block, and the first convolution layer of the i-th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by division, and obtains feature information after the convolution operation of each group of feature information, and then merges the number of channels of the feature information after the convolution operation of each group of feature information to obtain the output feature information of the first convolution layer.

[0171] For example, assuming that the number of channels of the i-1th feature information is 100, the i-1th feature information is divided into 10 groups, recorded as the first group of feature information, the second group of feature information, the third group of feature information...the tenth group of feature information, and the number of channels of each group is 10. Then, the first group of feature information, the second group of feature information, the third group of feature information...the tenth group of feature information are respectively subjected to standard convolution operations to obtain feature information 1 after the convolution operation of the first group of feature information, feature information 2 after the convolution operation of the second group of feature information, feature information 3 after the convolution operation of the third group of feature information...the feature information 10 after the convolution operation of the tenth group of feature information. Then, the above feature information 1, feature information 2, feature information 3...feature information 10 are merged by the number of channels, that is, the channel number is spliced ​​to obtain the output feature information of the first convolution layer.

[0172] From the above, it can be seen that the embodiment of the present application divides the convolution operation with a channel number of K (for example, 100) into G (for example, 10) convolution operations with a channel number of K / G (for example, 10) through grouped convolution. The amount of computation of the convolution operation with a channel number of K / G is much smaller than that of the convolution operation with a channel number of K. In this way, through grouped convolution, the amount of parameters and the amount of computation can be reduced to the original 1 / G, thereby effectively reducing the complexity of the convolution operation.

[0173] In one example, if Fig. 15BAs shown, the first convolution layer and the second convolution layer include grouped convolution layers, while the third convolution layer is not a grouped convolution layer. Assume that the input information of the first convolution layer and the second convolution layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this way, the filtering device inputs the i-1th feature information into the i-th residual block, and the first convolution layer of the i-th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by division, and obtains feature information after the convolution operation of each group of feature information, and then merges the number of channels of the feature information after the convolution operation of each group of feature information to obtain the output feature information of the first convolution layer. Similarly, the second convolutional layer of the i-th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by the division, and obtains feature information after the convolution operation on each group of feature information. Then, the number of channels of the feature information after the convolution operation on each group of feature information is merged to obtain the output feature information of the second convolutional layer.

[0174] In one example, if Fig. 15C As shown, the first convolution layer and the third convolution layer include grouped convolution layers, while the second convolution layer is not a grouped convolution layer. Assume that the input information of the first convolution layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this way, the filtering device inputs the i-1th feature information into the i-th residual block, and the first convolution layer of the i-th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by division, and obtains the feature information after the convolution operation of each group of feature information, and then merges the number of channels of the feature information after the convolution operation of each group of feature information to obtain the output feature information of the first convolution layer, which is recorded as the first feature information. At the same time, the i-1th feature information is convolved by the second convolution layer to obtain the second feature information. Then, the first feature information and the second feature information are spliced ​​to obtain the third feature information, and it is assumed that the number of channels of the third feature information is K1. The third feature information is input into the third convolutional layer. The third convolutional layer divides the third feature information into multiple groups based on the number of channels K1 of the third feature information. A standard convolution operation is performed on each of the multiple groups of feature information obtained by division to obtain feature information after the convolution operation on each group of feature information. Then, the number of channels of the feature information after the convolution operation on each group of feature information is merged to obtain the output feature information of the third convolutional layer.

[0175] In one example, if Fig.15DAs shown, the second convolution layer and the third convolution layer include grouped convolution layers, while the first convolution layer is not a grouped convolution layer. Assume that the input information of the second convolution layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this way, the filtering device inputs the i-1th feature information into the i-th residual block, and the first convolution layer of the i-th residual block performs a standard convolution operation on the i-1th feature information to obtain the first feature information. The second convolution layer of the i-th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by the division, and obtains the feature information after the convolution operation of each group of feature information. Then, the number of channels of the feature information after the convolution operation of each group of feature information is merged to obtain the output feature information of the second convolution layer, which is recorded as the second feature information. Then, the first feature information and the second feature information are spliced ​​to obtain the third feature information, and it is assumed that the number of channels of the third feature information is K1. The third feature information is input into the third convolutional layer. The third convolutional layer divides the third feature information into multiple groups based on the number of channels K1 of the third feature information. A standard convolution operation is performed on each of the multiple groups of feature information obtained by division to obtain feature information after the convolution operation on each group of feature information. Then, the number of channels of the feature information after the convolution operation on each group of feature information is merged to obtain the output feature information of the third convolutional layer.

[0176] In one example, if Fig.15EAs shown, the first convolution layer, the second convolution layer and the third convolution layer include a grouped convolution layer. Assume that the input information of the first convolution layer and the second convolution layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this way, the filtering device inputs the i-1th feature information into the i-th residual block, and the first convolution layer of the i-th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by division, and then merges the number of channels after obtaining the feature information after the convolution operation of each group of feature information, and obtains the first feature information output by the first convolution layer. The second convolution layer of the i-th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by division, and then merges the number of channels after obtaining the feature information after the convolution operation of each group of feature information, and obtains the second feature information output by the second convolution layer. Next, the first feature information and the second feature information are concatenated to obtain the third feature information, assuming that the number of channels of the third feature information is K1. The third feature information is input into the third convolutional layer, which divides the third feature information into multiple groups based on the number of channels K1 of the third feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by division, obtains feature information after the convolution operation of each group of feature information, and then merges the channels to obtain the output feature information of the third convolutional layer.

[0177] Above FIG. 15A to FIG. 15E The network structure of the i-th residual block when at least one of the first convolutional layer, the second convolutional layer, and the third convolutional layer includes a grouped convolutional layer is introduced.

[0178] In some embodiments, some of the convolutional layers in the first convolutional layer, the second convolutional layer, and the third convolutional layer include grouped convolutional layers, and some of the convolutional layers include sub-convolutional layers obtained by CP decomposition. For example, some of the convolutional layers in the first convolutional layer, the second convolutional layer, and the third convolutional layer include grouped convolutional layers, and at least one of the remaining convolutional layers includes a sub-convolutional layer obtained by tensor decomposition of a convolutional layer with a convolution kernel size of m×n, where m and n are both positive integers greater than 2.

[0179] In one example, the first convolution layer and / or the second convolution layer includes a grouped convolution layer, and the third convolution layer includes two sub-convolution layers obtained by tensor decomposition. Fig.16AAs shown in the figure, the first convolution layer is a grouped convolution layer with a convolution kernel size of 3X3, the second convolution layer is a convolution layer with a convolution kernel size of 1X1, and the third convolution layer includes a 1X3 sub-convolution layer and a 3X1 sub-convolution layer. In this way, by decomposing a 3X3 convolution operation into a 1X3 simple convolution operation and a 3X1 simple convolution operation, the computational complexity of the convolution operation can be greatly reduced, thereby improving the image filtering efficiency.

[0180] In one example, the third convolution layer and / or the second convolution layer includes a grouped convolution layer, and the first convolution layer includes two sub-convolution layers obtained by tensor decomposition. Fig. 16B As shown, the third convolution layer is a grouped convolution layer with a convolution kernel size of 3X3, the second convolution layer is a convolution layer with a convolution kernel size of 1X1, and the first convolution layer includes a 1X3 sub-convolution layer and a 3X1 sub-convolution layer.

[0181] In one example, the third convolutional layer and / or the first convolutional layer includes a grouped convolutional layer, and the second convolutional layer includes two sub-convolutional layers obtained by tensor decomposition. Fig. 16C As shown, the first convolution layer is a grouped convolution layer with a convolution kernel size of 3X3, the third convolution layer is a convolution layer with a convolution kernel size of 1X1, and the second convolution layer includes a 1X3 sub-convolution layer and a 3X1 sub-convolution layer.

[0182] The above example introduces a network structure in which one or two of the first convolutional layer, the second convolutional layer, and the third convolutional layer include a grouped convolutional layer, and the other convolutional layer includes two sub-convolutional layers obtained by tensor decomposition. In some embodiments, one of the first convolutional layer, the second convolutional layer, and the third convolutional layer includes a grouped convolutional layer, and the other two convolutional layers include a network structure in which two sub-convolutional layers are obtained by tensor decomposition.

[0183] For example, the second convolutional layer includes a grouped convolutional layer, and the first convolutional layer and the second convolutional layer each include two sub-convolutional layers obtained by tensor decomposition. Fig.16D As shown, the second convolution layer is a grouped convolution layer with a convolution kernel size of 1X1, the first convolution layer includes a 1X3 sub-convolution layer and a 3X1 sub-convolution layer, and the third convolution layer also includes a 1X3 sub-convolution layer and a 3X1 sub-convolution layer.

[0184] In this embodiment, the specific implementation methods of the first convolution layer, the second convolution layer, and some convolution layers in the third convolution layer in the i-th convolution kernel include grouped convolution layers, and some convolution layers include at least two sub-convolution layers obtained by CP decomposition, include but are not limited to the cases shown in the above examples, which include any other possibilities.

[0185] In some embodiments, for the j-th convolutional layer among the first convolutional layer, the second convolutional layer, and the third convolutional layer, as Fig.17A As shown, the j-th convolutional layer includes: decomposing and channel grouping the convolution layer with a convolution kernel size of m×n through tensor decomposition and grouped convolution, to obtain the first grouped sub-convolutional layer and the second grouped sub-convolutional layer, m and n are both positive integers greater than 2, and j is a positive integer.

[0186] The embodiment of the present application does not limit the convolution kernel sizes of the first group convolution layer and the second group convolution layer, as long as the convolution kernel sizes of the first group convolution layer and the second group convolution layer are both smaller than m×n.

[0187] In one example, the convolution kernel size of the first grouped sub-convolution layer is 1×n, and the convolution kernel size of the second grouped sub-convolution layer is m×1.

[0188] In this embodiment, at least one of the first convolution layer, the second convolution layer and the third convolution layer adopts both the tensor decomposition method and the grouped convolution method. For ease of description, the jth convolution layer is taken as an example for description.

[0189] In this embodiment, the convolution layer with a convolution kernel size of m×n is decomposed and the channels are grouped by tensor decomposition and grouped convolution to obtain a first grouped sub-convolution layer and a second grouped sub-convolution layer. The convolution kernel sizes of the first grouped sub-convolution layer and the second grouped sub-convolution layer are both less than m×n. Exemplarily, the convolution layer with a convolution kernel size of m×n can be first decomposed into two sub-convolution layers by tensor decomposition, recorded as the first sub-convolution layer and the second sub-convolution layer. Then, the first sub-convolution layer is convolved by grouped convolution to obtain a first grouped sub-convolution layer. The second sub-convolution layer is convolved by grouped convolution to obtain a second grouped sub-convolution layer. In the embodiment of the present application, the convolution operation of a larger convolution kernel is decomposed into a convolution operation of a smaller convolution kernel by tensor decomposition to reduce the computational replication of the convolution operation. At the same time, by grouping, the convolution operations with a large number of channels can be decomposed into convolution operations with a small number of channels, which can further reduce the computational complexity of the convolution operation and greatly improve the image filtering efficiency.

[0190] For example, assume that the number of channels of the input feature information of the jth convolutional layer is K, the convolution kernel size of the first grouped convolutional layer is 1×n, and the convolution kernel size of the second grouped convolutional layer is m×1. In this way, the input feature information with the number of channels K is input into the j convolutional layers, and the first grouped convolutional layer of the j convolutional layers divides the input feature information with the number of channels K into G groups, and performs a 1×n convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 1 output by the first grouped convolutional layer. Assuming that the number of channels of the feature information 1 is H, then, the feature information 1 with the number of channels H is input into the second grouped convolutional layer, and the feature information 1 with the number of channels H is divided into P groups by the second grouped convolutional layer, and after performing an m×1 convolution operation on each group of feature information, the number of channels is merged to obtain feature information 2 output by the second grouped convolutional layer.

[0191] The following is a collection of specific examples to introduce the network structure of the i-th residual block shown in this embodiment.

[0192] In one example, if Fig. 17B As shown, the first convolution layer includes a first grouped subconvolution layer and a second grouped subconvolution layer obtained by decomposing and grouping channels of a convolution layer with a convolution kernel size of m×n through a tensor decomposition method and a grouped convolution method, the convolution kernel size of the first grouped convolution layer is 1×n, and the convolution kernel size of the second grouped convolution layer is m×1. It is assumed that the input information of the first convolution layer and the second convolution layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this way, the filtering device inputs the feature information with the number of channels K into the first convolution layer, and the first grouped convolution layer in the first convolution layer divides the input feature information with the number of channels K into G groups, and performs an m×1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 1 output by the first grouped convolution layer. Assuming that the number of channels of the feature information 1 is H, then the feature information 1 with the number of channels H is input into the second grouped convolution layer, and the feature information 1 with the number of channels H is divided into P groups by the second grouped convolution layer, and each group of feature information is subjected to a 1×n convolution operation, and then the number of channels is merged to obtain the feature information 2 output by the second grouped convolution layer, and the first feature information output by the first convolution unit is obtained based on the feature information 2. The second convolution layer performs convolution and other operations on the i-1th feature information to obtain the second feature information output by the second convolution unit, and after merging the above first feature information and the second feature information, the convolution operation is input into the third convolution layer to obtain the output feature information of the i-th residual block.

[0193] For example Fig. 17CAs shown, the first convolution layer includes a first grouped sub-convolution layer with a convolution kernel size of 1×3 and a second grouped sub-convolution layer with a convolution kernel size of 3×1, which are decomposed and grouped by tensor decomposition and grouped convolution. The first grouped convolution layer divides the input feature information with a channel number of K into G groups, and performs a 3×1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 1 output by the first grouped convolution layer. Assuming that the number of channels of the feature information 1 is H, then the feature information 1 with a channel number of H is input into the second grouped convolution layer, and the feature information 1 with a channel number of H is divided into P groups by the second grouped convolution layer, and each group of feature information is subjected to a 1×3 convolution operation, and then the number of channels is merged to obtain feature information 2 output by the second grouped convolution layer. The feature information 2 is processed by the first activation function to obtain the first feature information output by the first convolution unit. After the second convolutional layer performs a convolution operation on the i-1th feature information, it is processed by the second activation function to obtain the second feature information output by the second convolution unit. After the first feature information and the second feature information are combined, they are input into the third convolutional layer for convolution and other operations to obtain the output feature information of the i-th residual block.

[0194] In one example, if Fig.17DAs shown, the first convolution layer includes a first grouped subconvolution layer and a second grouped subconvolution layer obtained by decomposing and grouping channels of a convolution layer with a convolution kernel size of m×n by tensor decomposition and grouped convolution, the convolution kernel size of the first grouped convolution layer is 1×n, the convolution kernel size of the second grouped convolution layer is m×1, and at least one of the second convolution layer and the third convolution layer includes a grouped convolution layer. Assume that the input information of the first convolution layer and the second convolution layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this way, the filtering device inputs the feature information with the number of channels K into the first convolution layer, and the first grouped convolution layer in the first convolution layer divides the input feature information with the number of channels K into G groups, and performs an m×1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 1 output by the first grouped convolution layer. Assuming that the number of channels of the feature information 1 is H, then the feature information 1 with the number of channels H is input into the second grouped convolution layer, and the feature information 1 with the number of channels H is divided into P groups by the second grouped convolution layer, and each group of feature information is subjected to a 1×n convolution operation, and then the number of channels is merged to obtain feature information 2 output by the second grouped convolution layer, and the first feature information output by the first convolution unit is obtained based on the feature information 2. The second convolution layer performs convolution (or grouped convolution) and other operations on the i-1th feature information to obtain the second feature information output by the second convolution unit, and after merging the above-mentioned first feature information and the second feature information, the convolution (or grouped convolution) and other operations are input into the third convolution layer to obtain the output feature information of the i-th residual block.

[0195] For example Fig.17EAs shown, the first convolution layer includes a first grouped sub-convolution layer with a convolution kernel size of 1×3 and a second grouped sub-convolution layer with a convolution kernel size of 3×1, which is decomposed and grouped by tensor decomposition and grouped convolution, and the second convolution layer and the second convolution layer include a grouped convolution layer. The first grouped convolution layer divides the input feature information with a channel number of K into G groups, and performs a 3×1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 1 output by the first grouped convolution layer. Assuming that the number of channels of the feature information 1 is H, then the feature information 1 with a channel number of H is input into the second grouped convolution layer, and the feature information 1 with a channel number of H is divided into P groups by the second grouped convolution layer, and each group of feature information is subjected to a 1×3 convolution operation, and then the number of channels is merged to obtain feature information 2 output by the second grouped convolution layer. The feature information 2 is processed by the first activation function to obtain the first feature information output by the first convolution unit. After the second convolutional layer performs a group convolution operation on the i-1th feature information, it is processed by the second activation function to obtain the second feature information output by the second convolution unit. After the first feature information and the second feature information are combined, they are input into the third convolutional layer for group convolution and other operations to obtain the output feature information of the i-th residual block.

[0196] In one example, if Fig.17FAs shown, the first convolution layer includes decomposing and grouping channels of a convolution layer with a convolution kernel size of m1×n1 by tensor decomposition and grouped convolution, and obtaining a grouped sub-convolution layer with a convolution kernel size of 1×n1 and a grouped sub-convolution layer with a convolution kernel size of m1×1. The third convolution layer includes decomposing and grouping channels of a convolution layer with a convolution kernel size of m2×n2 by tensor decomposition and grouped convolution, and obtaining a grouped sub-convolution layer with a convolution kernel size of 1×n2 and a grouped sub-convolution layer with a convolution kernel size of m2×1. Assume that the input information of the first convolution layer and the second convolution layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this way, the filtering device inputs the feature information with the number of channels K into the first convolution layer, and the grouped sub-convolution layer with the convolution kernel size of m1×1 in the first convolution layer divides the input feature information with the number of channels K into G groups, and performs an m1×1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 1 output by the grouped sub-convolution layer with the convolution kernel size of m1×1. Assuming that the number of channels of the feature information 1 is H, then, the feature information 1 with the number of channels H is input into the grouped sub-convolution layer with the convolution kernel size of 1×n1, and the grouped sub-convolution layer with the convolution kernel size of 1×n1 divides the feature information 1 with the number of channels H into P groups, and performs a 1×n1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 2 output by the grouped sub-convolution layer with the convolution kernel size of 1×n1, and the first feature information output by the first convolution unit is obtained based on the feature information 2. The second convolutional layer performs convolution (or grouped convolution) and other operations on the i-1th feature information to obtain the second feature information output by the second convolution unit, and merges the first feature information and the second feature information to obtain the third feature information, where the number of channels of the third feature information is assumed to be Q, and Q is a positive integer greater than 1. Next, the third feature information is input into the third convolutional layer, and the grouped sub-convolutional layer with a convolution kernel size of m2×1 in the third convolutional layer divides the third feature information with a channel number of Q into R groups, and performs an m2×1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 3 output by the grouped sub-convolutional layer with a convolution kernel size of m2×1. Assume that the feature information 3 has S channels. Then, the feature information 3 with S channels is input into the group sub-convolution layer with a convolution kernel size of 1×n2. The group sub-convolution layer with a convolution kernel size of 1×n2 divides the feature information 3 with S channels into T groups, and performs a 1×n2 convolution operation on each group of feature information. Then, the channels are merged to obtain feature information 4 output by the group sub-convolution layer with a convolution kernel size of 1×n2. Based on the feature information 4, the output feature information of the i-th residual block is obtained.

[0197] For example, Figure 17GAs shown, the first convolution layer includes a convolution layer with a convolution kernel size of 3×3, which is decomposed and channel grouped by tensor decomposition and grouped convolution, and a grouped sub-convolution layer with a convolution kernel size of 1×3 and a grouped sub-convolution layer with a convolution kernel size of 3×1 is obtained. The third convolution layer includes a convolution layer with a convolution kernel size of 3×3, which is decomposed and channel grouped by tensor decomposition and grouped convolution, and a grouped sub-convolution layer with a convolution kernel size of 1×3 and a grouped sub-convolution layer with a convolution kernel size of 3×1 is obtained. The second convolution layer includes a grouped convolution layer. Assume that the input information of the first convolution layer and the second convolution layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this way, the filtering device inputs the feature information with the number of channels K into the first convolution layer, and the grouped sub-convolution layer with the convolution kernel size of 3×1 in the first convolution layer divides the input feature information with the number of channels K into G groups, and performs a 3×1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 1 output by the grouped sub-convolution layer with the convolution kernel size of 3×1. Assuming that the number of channels of the feature information 1 is H, then, the feature information 1 with the number of channels H is input into the grouped sub-convolution layer with the convolution kernel size of 1×3, and the grouped sub-convolution layer with the convolution kernel size of 1×3 divides the feature information 1 with the number of channels H into P groups, and performs a 1×3 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 2 output by the grouped sub-convolution layer with the convolution kernel size of 1×3, and the first feature information output by the first convolution unit is obtained based on the feature information 2. The second convolutional layer performs convolution (or grouped convolution) and other operations on the i-1th feature information to obtain the second feature information output by the second convolution unit, and merges the first feature information and the second feature information to obtain the third feature information, where the number of channels of the third feature information is assumed to be Q, and Q is a positive integer greater than 1. Next, the third feature information is input into the third convolutional layer, and the grouped sub-convolutional layer with a convolution kernel size of 3×1 in the third convolutional layer divides the third feature information with a channel number of Q into R groups, and performs a 3×1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 3 output by the grouped sub-convolutional layer with a convolution kernel size of 3×1. Assume that the feature information 3 has S channels. Then, the feature information 3 with S channels is input into the grouped sub-convolutional layer with a convolution kernel size of 1×3. The grouped sub-convolutional layer with a convolution kernel size of 1×3 divides the feature information 3 with S channels into T groups, and performs a 1×3 convolution operation on each group of feature information. Then, the channels are merged to obtain feature information 4 output by the grouped sub-convolutional layer with a convolution kernel size of 1×3. Based on the feature information 4, the output feature information of the i-th residual block is obtained.

[0198] As can be seen from the above, in this embodiment, at least one of the first convolutional layer, the second convolutional layer, and the second convolutional layer in the above-mentioned i-th residual block includes a first grouped sub-convolutional layer and a second grouped sub-convolutional layer obtained by decomposing and channel grouping the convolutional layer with a convolution kernel size of m×n by tensor decomposition and grouped convolution. Optionally, the remaining convolutional layers in the first convolutional layer, the second convolutional layer, and the second convolutional layer include grouped convolutional layers. That is to say, based on the use of a multi-level receptive field method, the embodiment of the present application can simultaneously use CP decomposition and grouped convolution to replace at least one m×n standard convolutional layer (m, n are positive integers greater than or equal to 3) in an existing residual block, and / or can use grouped convolution to replace at least one of the remaining i×j standard convolutions in an existing residual block (i, j are positive integers).

[0199] In some embodiments, based on the above embodiments, Fig.18A As shown, the third convolution unit of the embodiment of the present application includes, in addition to the third convolution layer, a fourth convolution layer, and the fourth convolution layer is connected in series with the third convolution layer.

[0200] In some embodiments, the Fig.18B As shown, the fourth convolution layer includes a grouped convolution layer. For example, assuming that the feature information output by the third convolution layer is recorded as the fourth feature information, and the number of channels of the fourth feature information is recorded as K2, the filtering device inputs the fourth feature information output by the third convolution layer into the fourth convolution layer, and the fourth convolution layer divides the fourth feature information with the number of channels of K2 into multiple groups, and performs a standard convolution operation on each of the multiple groups of feature information obtained by division to obtain the feature information after the convolution operation of each group of feature information, and then, the feature information after the convolution operation of each group of feature information is merged by the number of channels to obtain the output feature information of the fourth convolution layer. Finally, the feature information output by the fourth convolution layer and the input feature information of the i-th residual block, such as the i-1-th feature information, are added to obtain the output feature information of the i-th residual block.

[0201] In some embodiments, Fig. 18C As shown in the figure, the third sub-convolution layer and the fourth sub-convolution layer are obtained by decomposing the convolution layer with a convolution kernel size of p×q through tensor decomposition, where p and q are both positive integers greater than 2.

[0202] The embodiment of the present application does not limit the convolution kernel sizes of the third sub-convolution layer and the fourth sub-convolution layer, as long as they are less than p×q.

[0203] In one example, the convolution kernel size of the third sub-convolution layer is 1×q, and the convolution kernel size of the fourth sub-convolution layer is p×1.

[0204] In this embodiment, if Fig. 18CAs shown, it is assumed that the feature information output by the third convolution layer is recorded as the fourth feature information. The filtering device inputs the fourth feature information output by the third convolution layer into the fourth convolution layer, and the third sub-convolution layer in the fourth convolution layer performs a simple 1×q convolution operation on the fourth feature information and outputs the fifth feature information. Next, the fifth feature information is input into the fourth sub-convolution layer for a simple p×1 convolution operation, and the sixth feature information is output. Finally, the sixth feature information output by the fourth convolution layer is added to the input feature information of the i-th residual block, for example, the i-1th feature information, to obtain the output feature information of the i-th residual block. In this way, the fourth convolution layer reduces the complexity of image filtering and improves the efficiency of image filtering by decomposing the complex p×q convolution operation into two simple 1×q and p×1 convolution operations.

[0205] In some embodiments, Fig.18D As shown, the fourth convolutional layer includes: decomposing and channel grouping the convolutional layer with a convolution kernel size of p×q through tensor decomposition and grouped convolution, to obtain a third grouped sub-convolutional layer and a fourth grouped sub-convolutional layer.

[0206] In this embodiment, if Fig.18D As shown, the feature information output by the third convolution layer is recorded as the fourth feature information. Assuming that the number of channels of the fourth feature information is K2, the convolution kernel size of the third grouping sub-convolution layer is 1×q, and the convolution kernel size of the fourth grouping sub-convolution layer is p×1. The filtering device inputs the fourth feature information output by the third convolution layer into the fourth convolution layer. The third grouping sub-convolution layer in the fourth convolution layer divides the fourth feature information with the number of channels K2 into multiple groups, and performs a p×1 convolution operation on each group of feature information, and then merges the number of channels to obtain feature information 4 output by the third grouping sub-convolution layer. Assume that the feature information 4 has a channel number K2. Then, the feature information 4 with the channel number K2 is input into the fourth grouping sub-convolution layer. The fourth grouping sub-convolution layer divides the feature information 4 with the channel number K2 into multiple groups, and performs a 1×q convolution operation on each group of feature information. Then, the channels are merged to obtain the feature information 5 output by the fourth grouping sub-convolution layer. Based on the feature information 5, the feature information output by the third convolution unit is obtained, and the feature information output by the third convolution unit is added to the input feature information of the i-th residual block, for example, the i-1-th feature information, to obtain the output feature information of the i-th residual block.

[0207] As can be seen from the above, the residual block provided in the embodiment of the present application includes a first convolution unit, a second convolution unit and a third convolution unit, wherein the first convolution unit, the second convolution unit and the third convolution unit respectively include at least one convolution layer. At least one convolution layer among the multiple convolution layers included in the residual block includes a grouped convolution layer formed by a grouped convolution method, and / or at least one convolution layer includes at least two grouped sub-convolution layers obtained by decomposing and channel grouping a convolution layer with a convolution kernel size of m×n by a tensor decomposition method and a grouped convolution method.

[0208] That is to say, the standard convolution layer in the existing residual block is decomposed into multiple convolution layers with lower computational complexity through the multi-level receptive field method and the grouped convolution method. Alternatively, the standard convolution layer in the existing residual block is decomposed into multiple convolution layers with lower computational complexity through the multi-level receptive field method, the grouped convolution method and the tensor decomposition method.

[0209] In one example, if Fig.19AAs shown, the i-th residual block includes a first convolution unit, a second convolution unit, and a third convolution unit, wherein the first convolution unit includes a grouped convolution layer (i.e., a first convolution layer) with a convolution kernel size of 3X3 and a first activation function (e.g., PReLU). The second convolution unit includes a second convolution layer with a convolution kernel size of 1X1 and a second activation function (e.g., PReLU). The third convolution unit includes a third convolution layer with a convolution kernel size of 1X1 and a grouped convolution layer (i.e., a fourth convolution layer) with a convolution kernel size of 3X3. The filtering device inputs the i-1th feature information into the i-th residual block, and the grouped convolution layer with a convolution kernel size of 3X3 in the first convolution unit in the i-th residual block divides the i-1th feature information into multiple groups, and performs a 3X3 convolution operation on each group of feature information, and then merges the number of channels to obtain the feature information output by the first convolution layer. Then, the feature information output by the first convolution layer is processed by the first activation function to obtain the first feature information output by the first convolution unit. After the second convolution layer with a convolution kernel size of 1X1 in the second convolution unit performs a standard convolution operation on the i-1th feature information, the feature information output by the second convolution layer is processed by the second activation function to obtain the second feature information output by the second convolution unit. Then, the first feature information and the second feature information are concatenated to obtain the third feature information, and the third feature information is input into the third convolution unit. The third convolution layer in the third convolution unit performs a standard convolution operation on the third feature information to obtain the fourth feature information, and the fourth feature information is input into the fourth convolution layer. The fourth convolution layer includes a grouped convolution layer with a convolution kernel size of 3X3, which divides the fourth feature information into multiple groups, and performs a 3X3 convolution operation on each group of feature information, and then merges the number of channels to obtain the feature information output by the third convolution unit. The feature information output by the third convolution unit is added to the i-1th feature information to obtain the feature information output by the i-th residual block, which can be recorded as the i-th feature information.

[0210] In one example, if Fig.19BAs shown, the i-th residual block includes a first convolution unit, a second convolution unit, and a third convolution unit, wherein the first convolution unit includes a first convolution layer with a convolution kernel size of 3X3 and a first activation function (e.g., PReLU). The second convolution unit includes a grouped convolution layer (i.e., the second convolution layer) with a convolution kernel size of 1X1 and a second activation function (e.g., PReLU). The third convolution unit includes a grouped convolution layer (i.e., the third convolution layer) with a convolution kernel size of 1X1 and a fourth convolution layer with a convolution kernel size of 3X3. The filtering device inputs the i-1th feature information into the i-th residual block, and the first convolution layer with a convolution kernel size of 3X3 in the first convolution unit in the i-th residual block performs a standard convolution operation on the i-1th feature information, and then processes the feature information output by the first convolution layer through the first activation function to obtain the first feature information output by the first convolution unit. The grouped convolution layer with a convolution kernel size of 3X3 in the second convolution unit in the i-th residual block divides the i-1th feature information into multiple groups, and performs a 3X3 convolution operation on each group of feature information, and then merges the number of channels to obtain the feature information output by the second convolution layer. Then, the feature information output by the second convolution layer is processed by the second activation function to obtain the second feature information output by the second convolution unit. Then, the first feature information and the second feature information are concatenated to obtain the third feature information, and the third feature information is input into the third convolution unit. The third convolution layer in the third convolution unit includes a grouped convolution layer with a convolution kernel size of 1X1, and the grouped convolution layer with a convolution kernel size of 1X1 divides the third feature information into multiple groups, and performs a 3X3 convolution operation on each group of feature information, and then merges the number of channels to obtain the fourth feature information output by the third convolution layer. Then, the fourth feature information is input into the fourth convolution layer for a standard convolution operation to obtain the feature information output by the third convolution unit. Finally, the feature information output by the third convolution unit is added to the i-1th feature information to obtain the feature information output by the i-th residual block, which can be recorded as the i-th feature information.

[0211] In one example, if Fig.19CAs shown, the i-th residual block includes a first convolution unit, a second convolution unit, and a third convolution unit, wherein the first convolution unit includes a first convolution layer and a first activation function (e.g., PReLU), wherein the first convolution layer includes a first grouped sub-convolution layer with a convolution kernel size of 1X3 and a second grouped sub-convolution layer with a convolution kernel size of 3X1. The second convolution unit includes a second convolution layer with a convolution kernel size of 1X1 and a second activation function (e.g., PReLU). The third convolution unit includes a third convolution layer with a convolution kernel size of 1X1 and a fourth convolution layer, wherein the fourth convolution layer includes a third grouped sub-convolution layer with a convolution kernel size of 1X3 and a fourth grouped sub-convolution layer with a convolution kernel size of 3X1. The filtering device inputs the i-1th feature information into the i-th residual block, and the first grouping subconvolution layer in the first convolution unit in the i-th residual block divides the i-1th feature information into multiple groups, and performs a 1X3 convolution operation on the feature information of each group and then merges the channels to obtain the output feature information of the first grouping subconvolution layer. Then, the output feature information of the first grouping subconvolution layer is input into the second grouping subconvolution layer for grouped 3X1 convolution to obtain the output feature information of the second grouping subconvolution layer. The output feature information of the second grouping subconvolution layer is processed by the first activation function to obtain the first feature information output by the first convolution unit. At the same time, the second convolution layer in the second convolution unit in the i-th residual block performs a 1X1 convolution operation on the i-1th feature information, and then processes the feature information output by the second convolution layer by the second activation function to obtain the second feature information output by the second convolution unit. Then, the first feature information and the second feature information are spliced ​​to obtain the third feature information, and the third feature information is input into the third convolution unit. The third convolution layer in the third convolution unit performs a 1X1 standard convolution operation on the third feature information to obtain the fourth feature information. Then, the fourth feature information is input into the fourth convolution layer. The fourth convolution layer includes a third grouping sub-convolution layer with a convolution kernel size of 1X3 and a fourth grouping sub-convolution layer with a convolution kernel size of 3X1, wherein the third grouping sub-convolution layer divides the fourth feature information into multiple groups, and performs a 1X3 convolution operation on the feature information of each group and then performs channel merging to obtain the output feature information of the third grouping sub-convolution layer. Then, the output feature information of the third grouping sub-convolution layer is input into the fourth grouping sub-convolution layer for grouped 3X1 convolution to obtain the output feature information of the fourth grouping sub-convolution layer. Finally, the feature information output by the third convolution unit is added to the i-1th feature information to obtain the feature information output by the i-th residual block, which can be recorded as the i-th feature information.

[0212] In one example, if Fig.19DAs shown, the i-th residual block includes a first convolution unit, a second convolution unit, and a third convolution unit, wherein the first convolution unit includes a first convolution layer and a first activation function (e.g., PReLU), wherein the first convolution layer includes a first grouped sub-convolution layer with a convolution kernel size of 1X3 and a second grouped sub-convolution layer with a convolution kernel size of 3X1. The second convolution unit includes a grouped convolution layer with a convolution kernel size of 1X1 and a second activation function (e.g., PReLU). The third convolution unit includes a grouped convolution layer with a convolution kernel size of 1X1 and a fourth convolution layer, the fourth convolution layer includes a third grouped sub-convolution layer with a convolution kernel size of 1X3 and a fourth grouped sub-convolution layer with a convolution kernel size of 3X1. The filtering device inputs the i-1th feature information into the i-th residual block, and the first grouping subconvolution layer in the first convolution unit in the i-th residual block divides the i-1th feature information into multiple groups, and performs a 1X3 convolution operation on the feature information of each group and then merges the channels to obtain the output feature information of the first grouping subconvolution layer. Then, the output feature information of the first grouping subconvolution layer is input into the second grouping subconvolution layer for grouped 3X1 convolution to obtain the output feature information of the second grouping subconvolution layer. The output feature information of the second grouping subconvolution layer is processed by the first activation function to obtain the first feature information output by the first convolution unit. At the same time, the second convolution layer in the second convolution unit in the i-th residual block performs a grouped 1X1 convolution operation on the i-1th feature information, and then processes the feature information output by the second convolution layer by the second activation function to obtain the second feature information output by the second convolution unit. Then, the first feature information and the second feature information are spliced ​​to obtain the third feature information, and the third feature information is input into the third convolution unit. The third convolution layer in the third convolution unit performs a grouped 1X1 convolution operation on the third feature information to obtain the fourth feature information. Then, the fourth feature information is input into the fourth convolution layer. The fourth convolution layer includes a third grouping sub-convolution layer with a convolution kernel size of 1X3 and a fourth grouping sub-convolution layer with a convolution kernel size of 3X1, wherein the third grouping sub-convolution layer divides the fourth feature information into multiple groups, and performs a 1X3 convolution operation on the feature information of each group and then performs channel merging to obtain the output feature information of the third grouping sub-convolution layer. Then, the output feature information of the third grouping sub-convolution layer is input into the fourth grouping sub-convolution layer for grouped 3X1 convolution to obtain the output feature information of the fourth grouping sub-convolution layer. Finally, the feature information output by the third convolution unit is added to the i-1th feature information to obtain the feature information output by the i-th residual block, which can be recorded as the i-th feature information.

[0213] The above embodiment introduces the process of obtaining the i-th residual block by processing the standard convolution layer based on the multi-level receptive field method and the grouped convolution method, and based on the multi-level receptive field method, the grouped convolution method and the tensor decomposition method in situation 1.

[0214] In case 2, the standard convolution layer is processed based on the group convolution method and the tensor decomposition method to obtain the i-th residual block.

[0215] Specifically, Fig. 20 As shown, the i-th residual block of the embodiment of the present application includes N convolutional layers connected in series, and the k-th convolutional layer among the N convolutional layers includes: decomposing and channel grouping the convolutional layer with a convolution kernel size of r×s through tensor decomposition and grouped convolution, to obtain the fifth grouped sub-convolutional layer and the sixth grouped sub-convolutional layer, r and s are both positive integers greater than 2, N is a positive integer, and k is a positive integer less than or equal to N.

[0216] The i-th residual block in the embodiment of the present application includes N convolutional layers connected in series, and some or all of the N convolutional layers include at least two sub-convolutional layers with low computational complexity obtained by decomposition through tensor decomposition and grouped convolution. In this way, the computational complexity of the residual block can be reduced without reducing the image processing performance of the residual block, thereby improving the image filtering performance of the neural network filter.

[0217] In one example, the embodiment of the present application can Fig. 7A The first 3×3 convolution layer in the residual block shown in the figure performs decomposition and channel grouping in the tensor decomposition method and the grouped convolution method, and obtains the grouped sub-convolution layer with a convolution kernel size of 1×3 and the grouped sub-convolution layer with a convolution kernel size of 3×1.

[0218] Optional, such as Fig.21A As shown, it can be Fig. 7A The second 3×3 convolutional layer in the residual block shown is not processed.

[0219] Optional, such as Fig. 21B As shown, Fig. 7A The second 3×3 convolutional layer in the residual block shown is decomposed into a 1×3 sub-convolutional layer and a 3×1 sub-convolutional layer by tensor decomposition.

[0220] Optional, such as Fig. 21C As shown, Fig. 7A The second 3×3 convolutional layer in the residual block shown is processed into a 3×3 grouped convolutional layer through grouped convolution.

[0221] Optional, such as Fig.21D As shown, the second 3×3 convolution layer in the residual block shown in Figure 7 is decomposed and channel grouped by tensor decomposition and grouped convolution to obtain a grouped sub-convolution layer with a convolution kernel size of 1×3 and a grouped sub-convolution layer with a convolution kernel size of 3×1.

[0222] In one example, the embodiment of the present application can Figure 7B The third 3×3 convolution layer in the residual block shown in the figure uses tensor decomposition and grouped convolution to decompose and group channels, and obtains a grouped sub-convolution layer with a convolution kernel size of 1×3 and a grouped sub-convolution layer with a convolution kernel size of 3×1.

[0223] Optionally, you can also Figure 7B The first and second 1×1 convolutional layers in the residual block shown are not processed.

[0224] Optionally, Figure 7B At least one of the first and second 1×1 convolutional layers in the residual block shown is decomposed into grouped convolutional layers by grouped convolution. For example, by grouped convolution, Figure 7B The first 1×1 convolution layer and the second 1×1 convolution layer in are both processed as grouped convolution layers.

[0225] The above embodiment introduces the process of processing the standard convolution layer based on the group convolution method and the tensor decomposition method in situation 2 to obtain the i-th residual block.

[0226] In some embodiments, Fig.22A As shown, the neural network filter of the embodiment of the present application also includes a shallow feature extraction unit, which includes at least one convolutional layer; the shallow feature extraction unit is used to extract shallow feature information of input information and input the shallow feature information into the residual unit.

[0227] Specifically, Fig.22A As shown, the neural network filter of the embodiment of the present application includes a shallow feature extraction unit and a residual unit. During actual filtering, the filtering device inputs the input information corresponding to the image to be filtered into the shallow feature extraction unit, and the shallow feature extraction unit extracts the shallow feature information of the input information. Then, the shallow feature information is input into the residual unit, and the residual unit re-extracts the shallow feature information to obtain the image feature information of the image to be filtered.

[0228] The shallow feature extraction unit of the embodiment of the present application Figure 6 The head of the neural network filter is shown.

[0229] In some embodiments, some convolutional layers in the shallow feature extraction unit of the present application embodiment are obtained by processing the standard convolutional layer in at least two ways among the multi-level receptive field method, the tensor decomposition method and the grouped convolution method. Figure 6The 3X3 convolution layer at the head of the neural network filter is decomposed by at least two of the multi-level receptive field method, the tensor decomposition method and the grouped convolution method to obtain multiple sub-convolution layers with low computational complexity. Figure 6 The 3X3 convolution layer at the head of the neural network filter shown is decomposed and grouped by tensor decomposition and group convolution to obtain two grouped sub-convolution layers. The specific decomposition process can refer to the relevant description of the above embodiment and will not be repeated here.

[0230] In some embodiments, Fig. 22B As shown, the neural network filter of the embodiment of the present application also includes a feature mapping unit, which includes at least one convolution layer; the feature mapping unit is used to map the image feature information output by the residual unit to obtain a filtered image.

[0231] Specifically, Fig. 22B As shown, the neural network filter of the embodiment of the present application includes a residual unit and a feature mapping unit. During actual filtering, the filtering device inputs the input information corresponding to the image to be filtered into the residual unit, and the residual unit extracts the feature information of the input information to obtain the image feature information of the image to be filtered. Then, the image feature information is input into the feature mapping unit, and the feature mapping unit performs mapping processing on the feature mapping unit to obtain a filtered image of the image to be filtered.

[0232] In one example, if Fig. 22C As shown, the neural network filter of the embodiment of the present application includes a shallow feature extraction unit, a residual unit and a feature mapping unit. During actual filtering, the filtering device inputs the input information corresponding to the image to be filtered into the shallow feature extraction unit, and the shallow feature extraction unit extracts the shallow feature information of the input information. Then, the shallow feature information is input into the residual unit, and the residual unit performs feature re-extraction on the shallow feature information to obtain the image feature information of the image to be filtered. Finally, the image feature information is input into the feature mapping unit for mapping processing to obtain a filtered image.

[0233] The feature mapping unit of the embodiment of the present application Figure 6 The tail of the neural network filter shown corresponds to .

[0234] In some embodiments, some convolutional layers in the feature mapping unit of the present application embodiment are obtained by processing the standard convolutional layer in at least two ways among the multi-level receptive field method, the tensor decomposition method and the grouped convolution method. Figure 6 The 3X3 convolution layer at the tail of the neural network filter is decomposed by at least two of the multi-level receptive field method, the tensor decomposition method and the grouped convolution method to obtain multiple sub-convolution layers with low computational complexity. Figure 6 The 3X3 convolution layer at the tail of the neural network filter is decomposed and grouped by tensor decomposition and group convolution to obtain two grouped sub-convolution layers. The specific decomposition process can refer to the relevant description of the above embodiment and will not be repeated here.

[0235] In some embodiments, the neural network filter of the embodiment of the present application may be a single-branch convolutional neural network, which may perform filtering processing on the brightness component and the chrominance component of the image to be filtered.

[0236] In some embodiments, the neural network filter of the embodiment of the present application may be a two-branch convolutional neural network that performs separate processing on the luminance component and the chrominance component. Fig.23A As shown, the residual unit of the embodiment of the present application includes a luminance component residual unit and a chrominance component residual unit, wherein the luminance component residual unit is used to perform feature extraction processing on the luminance component of the image to be filtered to obtain image feature information of the luminance component of the image to be filtered. The chrominance component residual unit is used to perform feature extraction processing on the chrominance component of the image to be filtered to obtain image feature information of the chrominance component of the image to be filtered.

[0237] In some embodiments, if the neural network filter includes a feature mapping unit, such as Fig. 23B As shown, the feature mapping unit includes a luminance component feature mapping unit connected to the luminance component residual unit, and a chrominance component feature mapping unit connected to the chrominance component residual unit.

[0238] Among them, the brightness component feature mapping unit is used to map the image brightness feature information output by the brightness component residual unit; the chrominance component feature mapping unit is used to map the image chrominance feature information output by the chrominance component residual unit.

[0239] In some embodiments, Fig.23C As shown, the neural network filter also includes a shallow feature extraction unit. Fig.23CAs shown, during actual filtering, the input information corresponding to the image to be filtered is input into the shallow feature extraction unit, and the shallow feature extraction unit extracts shallow feature information from the input information to obtain shallow feature information of the input information. Next, the shallow feature information of the brightness component in the shallow feature information is input into the brightness component residual unit, and the brightness component residual unit performs feature extraction processing on the shallow feature information of the brightness component to obtain image feature information of the brightness component of the image to be filtered. Then, the image feature information of the brightness component of the image to be filtered is input into the brightness component feature mapping unit for mapping processing to obtain the filtered image information of the brightness component of the image to be filtered. Similarly, the shallow feature information of the chroma component in the shallow feature information is input into the chroma component residual unit, and the chroma component residual unit performs feature extraction processing on the shallow feature information of the chroma component to obtain the image feature information of the chroma component of the image to be filtered. Then, the image feature information of the chroma component of the image to be filtered is input into the chroma component feature mapping unit for mapping processing to obtain the filtered image information of the chroma component of the image to be filtered. Finally, the filtered image information of the brightness component and the filtered image information of the chrominance component of the image to be filtered are added to the input information to obtain a filtered image of the image to be filtered.

[0240] In some embodiments, if the image to be filtered is a reconstructed image, the filtering method of the embodiment of the present application can be applied to video post-processing and video encoding and decoding.

[0241] For example, if used in video encoding and decoding, the filtering device can directly input the filtered image of the reconstructed image into the display device for display, and can also save the filtered image of the reconstructed image into the decoded image cache as a reference image for subsequent other images to predict other images.

[0242] If applied to video post-processing, the filtered image of the reconstructed image can be directly input into a display device for display, instead of being used as a reference image for other images.

[0243] The neural network-based image filtering method proposed in the embodiment of the present application proposes a new neural network filter, which includes multiple convolutional layers for extracting image feature information. The multiple convolutional layers are obtained by processing the standard convolutional layers in at least two of the multi-level receptive field method, the tensor decomposition method, and the grouped convolution method. For example, the standard convolutional layer in the existing neural network filter, such as Figure 6 or Figure 8The standard convolutional layers (e.g., 3x3 convolutional layers) in the head, main body, tail, etc. of the neural network filter shown are processed by at least two of the multi-level receptive field method, the tensor decomposition method, and the grouped convolution method, and converted into convolution operations with low computational complexity, thereby reducing the filtering operation complexity of the neural network filter without reducing the filtering performance of the neural network filter in advance, thereby improving the filtering effect of the image.

[0244] The above describes the image filtering method and the network structure of the neural network filter proposed in the embodiment of the present application. The neural network filter described in the above embodiment can be applied to video decoding.

[0245] Fig.24 This is a flow chart of an image decoding method provided in an embodiment of the present application. The method in the embodiment of the present application is applied to a decoding device or a decoder. Fig.24 As shown, including:

[0246] S201, decoding a code stream, determining a reconstructed image of a current image, and determining input information of a neural network filter based on the reconstructed image.

[0247] The input information includes a reconstructed image.

[0248] The method of the embodiment of the present application can be applied to a video decoding solution or to video post-processing.

[0249] In some embodiments, when applied to a video decoding scheme, the encoder first divides the current image into coding blocks when encoding the current image, and performs block-by-block encoding using the coding blocks as coding units. For example, for the current block to be encoded in the current image, first obtain the prediction value of the current block through inter-frame and / or intra-frame prediction. Then, based on the prediction value of the current block and the current block, obtain the residual value of the current block. The encoder transforms the residual value of the current block to obtain a transformation coefficient. In one example, the encoder does not quantize the transformation coefficient of the current block, but directly encodes the transformation coefficient to obtain a code stream. In another example, the encoder quantizes the transformation coefficient of the current block to obtain a quantization coefficient, and then encodes the quantization coefficient to obtain a code stream.

[0250] During the encoding process, Figure 2As shown, the encoding end also performs an inverse transformation on the transform coefficients to obtain a residual value, and adds the residual value and the predicted value to obtain a reconstruction value of the current block. Based on the above steps, the reconstruction value of each coding block in the current image can be obtained, and these reconstruction values ​​constitute the reconstructed image of the current image. Then, in order to further improve the quality of the reconstructed image, the reconstructed image is filtered to obtain a decoded image of the current image. In one example, the decoded image can be stored in a decoded image cache for prediction of subsequent images.

[0251] like Figure 3 As shown, for each block to be decoded in the current image, such as the current block, after the decoder obtains the code stream, it decodes the code stream to obtain the transform coefficient of the current block. In one example, if the encoder quantizes the transform coefficient and then encodes it, the decoder decodes the code stream, obtains the quantization coefficient of the current block, and then dequantizes the quantization coefficient to obtain the transform coefficient of the current image. Then, the decoder performs a decompression on the transform coefficient of the current block to obtain the residual value of the current block. At the same time, the decoder uses inter-frame and / or intra-frame prediction to predict the predicted value of the current block. In this way, the predicted value and the residual value of the current block are added to obtain the reconstruction value of the current block. Based on the above steps, the decoder can decode and determine the reconstruction value of each block to be decoded in the current image, and these reconstruction values ​​constitute the reconstructed image of the current image. Then, in order to further improve the quality of the reconstructed image, the decoder filters the reconstructed image to obtain the decoded image of the current image. In one example, the decoder can store the decoded image in a decoded image cache for prediction of subsequent images. In one example, the decoder can output the decoded image to a display device for display.

[0252] In some embodiments, the image filtering method proposed in the embodiments of the present application can be used to filter at least one frame of image in a video. That is, the current image is an image in the video.

[0253] In some embodiments, the image filtering method proposed in the embodiments of the present application can be used to decode a single image, that is, the current image is a single image, such as an image generated by an electronic device.

[0254] In an embodiment of the present application, the decoding end uses a neural network filter to filter the reconstructed image.

[0255] When filtering the reconstructed image, the embodiment of the present application can directly filter the entire reconstructed image as a filtering unit, or divide the reconstructed image into multiple blocks, and filter each block as a filtering unit. For ease of description, the embodiment of the present application uses the reconstructed image as a filtering unit, and the reconstructed image can be a reconstructed image (i.e., a frame) or a partial area of ​​the reconstructed image.

[0256] In a possible implementation manner, the reconstructed image includes at least one CTU.

[0257] In another possible implementation manner, the reconstructed image includes a preset image area of ​​a frame of reconstructed image.

[0258] The embodiment of the present application does not limit the specific method of determining the input information of the neural network filter based on the reconstructed image.

[0259] In a possible implementation manner, the above input information only includes the reconstructed image.

[0260] In a possible implementation, the input information includes, in addition to the reconstructed image, at least one of a predicted image corresponding to the reconstructed image, a boundary strength image, a frame type (slice type) of the reconstructed image, and quantization parameter information.

[0261] S202, filtering the input information through a neural network filter to obtain a filtered image of the reconstructed image.

[0262] Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and these multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

[0263] Among them, the specific network structure of the neural network filter is described with reference to the above embodiment and will not be repeated here.

[0264] In some embodiments, the image filtering method of the embodiment of the present application can be used in a video decoder, for example, in loop filtering, where the filtered reconstructed image is stored in a decoded image cache for subsequent image decoding, and the filter output affects the video decoding. Since the embodiment of the present application provides improved filtering quality of the reconstructed image, the decoding effect of the video can be improved when subsequent decoding is performed based on the reconstructed image with better quality.

[0265] In some embodiments, the image filtering method of the embodiment of the present application can be used in a video post-processing product. For example, it can be applied to a display end, that is, the filtered reconstructed image is directly displayed without being placed in a decoded image cache, and the filter output does not affect video decoding.

[0266] In some embodiments, the image decoding method of the embodiments of the present application can also be used in any module that uses a neural network in neural network-based video encoding, such as a neural network super-resolution module, a neural network intra-frame prediction module, etc., by simply replacing the reconstructed image block with the corresponding image block.

[0267] The image filtering method provided in the embodiment of the present application is that the decoding end determines the reconstructed image of the current image by decoding the code stream, and determines the input information of the neural network filter based on the reconstructed image, and the input information includes the reconstructed image; the input information is filtered by the neural network filter to obtain the filtered image of the reconstructed image; wherein the neural network filter includes multiple convolution layers for extracting image feature information, and the multiple convolution layers are obtained by processing the standard convolution layers by at least two methods including the multi-level receptive field method, the tensor decomposition method and the grouped convolution method. That is, in the embodiment of the present application, the existing standardized convolution operation is decomposed into a convolution operation with low computational complexity by at least two methods including the multi-level receptive field method, the tensor decomposition method and the grouped convolution method, thereby reducing the filtering operation complexity of the neural network filter without reducing the filtering performance of the neural network filter in advance, thereby improving the filtering effect of the image.

[0268] The above describes the image filtering method of the embodiment of the present application by taking the decoding end as an example. The following describes the image filtering method of the embodiment of the present application by taking the encoding end as an example.

[0269] Fig.25 A flowchart of an image filtering method provided in an embodiment of the present application is shown in FIG. Figure 1 or Figure 2 The encoder or other electronic device shown in the figure is used as an example for convenience of description. Fig.25 As shown, the method of the embodiment of the present application includes:

[0270] S301. Encode the current image, determine a reconstructed image of the current image, and determine input information of a neural network filter based on the reconstructed image.

[0271] The input information includes a reconstructed image.

[0272] The method of the embodiment of the present application can be applied to a video encoding scheme, and can also be applied to video post-processing.

[0273] In some embodiments, when applied to a video coding scheme, the encoder first divides the current image into coding blocks when encoding the current image, and performs block-by-block encoding using the coding blocks as coding units. For example, for the current block to be encoded in the current image, first obtain the prediction value of the current block through inter-frame and / or intra-frame prediction. Then, based on the prediction value of the current block and the current block, obtain the residual value of the current block. The encoder transforms the residual value of the current block to obtain a transformation coefficient. In one example, the encoder does not quantize the transformation coefficient of the current block, but directly encodes the transformation coefficient to obtain a code stream. In another example, the encoder quantizes the transformation coefficient of the current block to obtain a quantization coefficient, and then encodes the quantization coefficient to obtain a code stream.

[0274] During the encoding process, Figure 2 As shown, the encoding end also performs an inverse transformation on the transform coefficients to obtain a residual value, and adds the residual value and the predicted value to obtain a reconstruction value of the current block. Based on the above steps, the reconstruction value of each coding block in the current image can be obtained, and these reconstruction values ​​constitute the reconstructed image of the current image. Then, in order to further improve the quality of the reconstructed image, the reconstructed image is filtered to obtain a decoded image of the current image. In one example, the decoded image can be stored in a decoded image cache for prediction of subsequent images.

[0275] In some embodiments, the image filtering method proposed in the embodiments of the present application can be used to filter at least one frame of image in a video. That is, the current image is an image in the video.

[0276] In some embodiments, the image filtering method proposed in the embodiments of the present application can be used to encode a single image, that is, the current image is a single image, such as an image generated by an electronic device.

[0277] In an embodiment of the present application, the encoding end uses a neural network filter to filter the reconstructed image.

[0278] When filtering the reconstructed image, the embodiment of the present application can directly filter the entire reconstructed image as a filtering unit, or divide the reconstructed image into multiple blocks, and filter each block as a filtering unit. For ease of description, the embodiment of the present application uses the reconstructed image as a filtering unit, and the reconstructed image can be a reconstructed image (i.e., a frame) or a partial area of ​​the reconstructed image.

[0279] In a possible implementation manner, the reconstructed image includes at least one CTU.

[0280] In another possible implementation manner, the reconstructed image includes a preset image area of ​​a frame of reconstructed image.

[0281] The embodiment of the present application does not limit the specific method of determining the input information of the neural network filter based on the reconstructed image.

[0282] In a possible implementation manner, the above input information only includes the reconstructed image.

[0283] In a possible implementation, the input information includes, in addition to the reconstructed image, at least one of a predicted image corresponding to the reconstructed image, a boundary strength image, a frame type (slice type) of the reconstructed image, and quantization parameter information.

[0284] S302, filtering the input information through a neural network filter to obtain a filtered image of the reconstructed image.

[0285] Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and these multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

[0286] Among them, the specific network structure of the neural network filter is described with reference to the above embodiment and will not be repeated here.

[0287] In some embodiments, the image filtering method of the embodiment of the present application can be used in a video encoder, for example, in loop filtering, where the filtered reconstructed image is stored in a decoded image cache for subsequent image encoding, and the filter output affects the video encoding. Since the embodiment of the present application provides improved filtering quality of the reconstructed image, the encoding effect of the video can be improved when subsequent encoding is performed based on the reconstructed image with better quality.

[0288] In some embodiments, the image filtering method of the embodiment of the present application can be used in a video post-processing product. For example, it can be applied to a display end, that is, the filtered reconstructed image is directly displayed without being placed in a decoded image cache, and the filter output does not affect the video encoding.

[0289] In some embodiments, the image encoding method of the embodiments of the present application can also be used in any module that uses a neural network in neural network-based video encoding, such as a neural network super-resolution module, a neural network intra-frame prediction module, etc., by simply replacing the reconstructed image block with the corresponding image block.

[0290] The image filtering method provided in the embodiment of the present application is that the encoding end determines the reconstructed image of the current image by encoding the current image, and determines the input information of the neural network filter based on the reconstructed image, and the input information includes the reconstructed image; the input information is filtered by the neural network filter to obtain the filtered image of the reconstructed image; wherein the neural network filter includes multiple convolution layers for extracting image feature information, and the multiple convolution layers are obtained by processing the standard convolution layers by at least two methods including the multi-level receptive field method, the tensor decomposition method and the grouped convolution method. That is, in the embodiment of the present application, the existing standardized convolution operation is decomposed into a convolution operation with low computational complexity by at least two methods including the multi-level receptive field method, the tensor decomposition method and the grouped convolution method, thereby reducing the filtering operation complexity of the neural network filter without reducing the filtering performance of the neural network filter in advance, thereby improving the filtering effect of the image.

[0291] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0292] It should also be understood that in the various method embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0293] Combination of the above Figures 9 to 25 , describes in detail the method embodiment of the present application, and the following is combined with Figure 26 to Figure 28 , describe in detail the device embodiments of the present application.

[0294] Fig.26 It is a schematic block diagram of a neural network-based image filtering device provided in one embodiment of the present application.

[0295] like Fig.26 As shown, the neural network-based image filtering device 10 may include:

[0296] A determination unit 11, configured to determine input information of a neural network filter based on an image to be filtered, wherein the input information includes the image to be filtered;

[0297] The filtering unit 12 is used to filter the input information through a neural network filter to obtain a filtered image of the image to be filtered; wherein the neural network filter includes a plurality of convolutional layers for extracting image feature information, and the plurality of convolutional layers are obtained by processing the standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method, and a grouped convolution method.

[0298] In some embodiments, the neural network filter includes a residual unit, the residual unit includes a plurality of residual blocks connected in sequence, the i-th residual block among the plurality of residual blocks includes all or part of the convolutional layers among the plurality of convolutional layers, and i is a positive integer.

[0299] In some embodiments, the i-th residual block includes a first convolution unit, a second convolution unit, and a third convolution unit. The first convolution unit and the second convolution unit are connected in parallel and then connected in series with the third convolution unit. The first convolution unit includes a first convolution layer, the second convolution unit includes a second convolution layer, and the third convolution unit includes a third convolution layer. The convolution kernel sizes of the first convolution layer and the second convolution layer are different, and at least one of the first convolution layer, the second convolution layer and the third convolution layer includes a grouped convolution layer. The grouped convolution layer is used to divide the input feature information of the grouped convolution layer into multiple groups based on the number of channels through the grouped convolution method, and perform convolution operations on each group of feature information and then merge the number of channels as the output information of the grouped convolution layer.

[0300] In some embodiments, for the j-th convolutional layer among the first convolutional layer, the second convolutional layer and the third convolutional layer, the j-th convolutional layer includes: decomposing and channel grouping the convolutional layer with a convolution kernel size of m×n through the tensor decomposition method and the grouped convolution method, to obtain a first grouped sub-convolutional layer and a second grouped sub-convolutional layer, where m and n are both positive integers greater than 2, and j is a positive integer.

[0301] In some embodiments, the convolution kernel size of the first grouped sub-convolution layer is 1×n, and the convolution kernel size of the second grouped sub-convolution layer is m×1.

[0302] In some embodiments, the third convolution unit further includes a fourth convolution layer, and the fourth convolution layer is connected in series with the third convolution layer.

[0303] In some embodiments, the fourth convolutional layer includes a grouped convolutional layer; or, the fourth convolutional layer includes: a third sub-convolutional layer and a fourth sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of p×q by the tensor decomposition method, where p and q are both positive integers greater than 2; or, the fourth convolutional layer includes: a third grouped sub-convolutional layer and a fourth grouped sub-convolutional layer obtained by decomposing and channel grouping a convolutional layer with a convolution kernel size of p×q by the tensor decomposition method and the grouped convolution method.

[0304] In some embodiments, the i-th residual block includes N convolutional layers connected in series, and the k-th convolutional layer among the N convolutional layers includes: decomposing and channel grouping the convolutional layer with a convolution kernel size of r×s through the tensor decomposition method and the grouped convolution method, to obtain the fifth grouped sub-convolutional layer and the sixth grouped sub-convolutional layer, wherein r and s are both positive integers greater than 2, N is a positive integer, and k is a positive integer less than or equal to N.

[0305] In some embodiments, the neural network filter also includes: a shallow feature extraction unit, the shallow feature extraction unit includes at least one convolutional layer; the shallow feature extraction unit is used to extract shallow feature information of the input information and input the shallow feature information into the residual unit.

[0306] In some embodiments, the neural network filter further includes: a feature mapping unit, the feature mapping unit including at least one convolutional layer; the feature mapping unit is used to map the image feature information output by the residual unit to obtain the filtered image.

[0307] In some embodiments, the residual unit includes a luminance component residual unit and a chrominance component residual unit. The luminance component residual unit is used to perform feature extraction processing on the luminance component of the image to be filtered to obtain image feature information of the luminance component of the image to be filtered; the chrominance component residual unit is used to perform feature extraction processing on the chrominance component of the image to be filtered to obtain image feature information of the chrominance component of the image to be filtered.

[0308] In some embodiments, the neural network filter also includes a luminance component feature mapping unit connected to the luminance component residual unit, and a chrominance component feature mapping unit connected to the chrominance component residual unit; the luminance component feature mapping unit is used to map the image luminance feature information output by the luminance component residual unit; the chrominance component feature mapping unit is used to map the image chrominance feature information output by the chrominance component residual unit.

[0309] In some embodiments, the image to be filtered is a reconstructed image, and the filtering unit 12 is further used to input the filtered image of the reconstructed image into a display device for display, and save the filtered image of the reconstructed image into a decoded image cache as a reference image for other images; or, input the filtered image of the reconstructed image into a display device for display, and not as a reference image.

[0310] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig.26 The device shown can execute the above-mentioned filtering method embodiment, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the method embodiment corresponding to the filtering device, which will not be described in detail here for the sake of brevity.

[0311] Fig. 27 It is a schematic block diagram of an image decoding device provided in one embodiment of the present application.

[0312] like Fig. 27 As shown, the image decoding device 20 may include:

[0313] A decoding unit 21, configured to decode a bitstream, determine a reconstructed image of a current image, and determine input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image;

[0314] The filtering unit 22 is used to filter the input information through a neural network filter to obtain a filtered image of the reconstructed image; wherein the neural network filter includes a plurality of convolutional layers for extracting image feature information, and the plurality of convolutional layers are obtained by processing the standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method, and a grouped convolution method.

[0315] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig. 27 The device shown can execute the embodiments of the above method, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the method embodiments corresponding to the decoder, which will not be described in detail here for the sake of brevity.

[0316] Fig.28 It is a schematic block diagram of an image encoding device provided by an embodiment of the present application.

[0317] like Fig.28 As shown, the image encoding device 30 may include:

[0318] The encoding unit 31 is used to encode the current image, determine a reconstructed image of the current image, and determine input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image;

[0319] The filtering unit 32 is used to filter the input information through a neural network filter to obtain a filtered image of the reconstructed image; wherein the neural network filter includes a plurality of convolutional layers for extracting image feature information, and the plurality of convolutional layers are obtained by processing the standard convolutional layers through at least two methods of a multi-level receptive field method, a tensor decomposition method, and a grouped convolution method.

[0320] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig.28 The device shown can execute the embodiments of the above method, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the method embodiments corresponding to the encoder, which will not be described in detail here for the sake of brevity.

[0321] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0322] Fig.29 is a schematic block diagram of an electronic device provided in an embodiment of the present application, Fig.29 The electronic device may be the above-mentioned decoder or encoder or filtering device.

[0323] like Fig.29 As shown, the electronic device 30 may include:

[0324] The memory 31 and the processor 32, the memory 31 is used to store the computer program 33 and transmit the program code 33 to the processor 32. In other words, the processor 32 can call and run the computer program 33 from the memory 31 to implement the method in the embodiment of the present application.

[0325] For example, the processor 32 may be configured to execute the steps in the above method according to the instructions in the computer program 33 .

[0326] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0327] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0328] In some embodiments of the present application, the memory 31 includes but is not limited to:

[0329] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0330] In some embodiments of the present application, the computer program 33 may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to complete the method for recording pages provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 33 in the electronic device.

[0331] like Fig.29 As shown, the electronic device 30 may further include:

[0332] The transceiver 34 may be connected to the processor 32 or the memory 31 .

[0333] The processor 32 may control the transceiver 34 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 34 may include a transmitter and a receiver. The transceiver 34 may further include an antenna, and the number of antennas may be one or more.

[0334] It should be understood that the various components in the electronic device 30 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0335] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to perform the method of the above method embodiment.

[0336] The embodiment of the present application also provides a computer program product including instructions, which, when executed by a computer, enables the computer to execute the method of the above method embodiment.

[0337] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above method embodiment.

[0338] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function of the embodiment of the present application is implemented in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0339] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0340] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0341] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0342] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A neural network-based image filtering method, characterized in that: include: Based on the image to be filtered, determining input information of the neural network filter, the input information including the image to be filtered; The input information is filtered by a neural network filter to obtain a filtered image of the image to be filtered; Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

2. The method according to claim 1, characterized in that The neural network filter includes a residual unit, the residual unit includes a plurality of residual blocks connected in sequence, the i-th residual block among the plurality of residual blocks includes all or part of the convolutional layers among the plurality of convolutional layers, and i is a positive integer.

3. The method according to claim 2, characterized in that The i-th residual block includes a first convolution unit, a second convolution unit, and a third convolution unit, wherein the first convolution unit and the second convolution unit are connected in parallel and then connected in series with the third convolution unit; The first convolution unit includes a first convolution layer, the second convolution unit includes a second convolution layer, the third convolution unit includes a third convolution layer, the convolution kernel sizes of the first convolution layer and the second convolution layer are different, and at least one of the first convolution layer, the second convolution layer and the third convolution layer includes a grouped convolution layer, and the grouped convolution layer is used to divide the input feature information of the grouped convolution layer into multiple groups based on the number of channels through the grouped convolution method, and perform convolution operations on each group of feature information respectively and then merge the number of channels as the output information of the grouped convolution layer.

4. The method according to claim 3, characterized in that For the j-th convolutional layer among the first convolutional layer, the second convolutional layer and the third convolutional layer, the j-th convolutional layer includes: decomposing and channel grouping the convolutional layer with a convolution kernel size of m×n through the tensor decomposition method and the grouped convolution method, to obtain a first grouped sub-convolutional layer and a second grouped sub-convolutional layer, wherein m and n are both positive integers greater than 2, and j is a positive integer.

5. The method according to claim 4, characterized in that The convolution kernel size of the first grouped sub-convolution layer is 1×n, and the convolution kernel size of the second grouped sub-convolution layer is m×1.

6. The method according to any one of claims 3 to 5, characterized in that: The third convolution unit also includes a fourth convolution layer, and the fourth convolution layer is connected in series with the third convolution layer.

7. The method according to claim 6, characterized in that The fourth convolutional layer comprises a grouped convolutional layer; or, The fourth convolutional layer includes: a third sub-convolutional layer and a fourth sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of p×q by the tensor decomposition method, wherein p and q are both positive integers greater than 2; or, The fourth convolutional layer includes: decomposing and channel grouping the convolutional layer with a convolution kernel size of p×q through the tensor decomposition method and the grouped convolution method, to obtain a third grouped sub-convolutional layer and a fourth grouped sub-convolutional layer.

8. The method according to claim 2, characterized in that: The i-th residual block includes N convolutional layers connected in series, and the k-th convolutional layer among the N convolutional layers includes: decomposing and channel grouping the convolutional layer with a convolution kernel size of r×s through the tensor decomposition method and the grouped convolution method, to obtain the fifth grouped sub-convolutional layer and the sixth grouped sub-convolutional layer, wherein r and s are both positive integers greater than 2, N is a positive integer, and k is a positive integer less than or equal to N.

9. The method according to claim 2, characterized in that: The neural network filter further comprises: a shallow feature extraction unit, wherein the shallow feature extraction unit comprises at least one convolutional layer; The shallow feature extraction unit is used to extract shallow feature information of the input information and input the shallow feature information into the residual unit.

10. The method according to claim 2, characterized in that The neural network filter further comprises: a feature mapping unit, wherein the feature mapping unit comprises at least one convolutional layer; The feature mapping unit is used to perform mapping processing on the image feature information output by the residual unit to obtain the filtered image.

11. The method according to any one of claims 2-5, 7-10, characterized in that: The residual unit includes a luminance component residual unit and a chrominance component residual unit, The brightness component residual unit is used to perform feature extraction processing on the brightness component of the image to be filtered to obtain image feature information of the brightness component of the image to be filtered; The chrominance component residual unit is used to perform feature extraction processing on the chrominance component of the image to be filtered to obtain image feature information of the chrominance component of the image to be filtered.

12. The method according to claim 11, characterized in that The neural network filter also includes a luminance component feature mapping unit connected to the luminance component residual unit, and a chrominance component feature mapping unit connected to the chrominance component residual unit; The brightness component feature mapping unit is used to map the image brightness feature information output by the brightness component residual unit; The chrominance component feature mapping unit is used to perform mapping processing on the image chrominance feature information output by the chrominance component residual unit.

13. The method according to claim 1, characterized in that The image to be filtered is a reconstructed image, and the method further includes: The filtered image of the reconstructed image is input into a display device for display, and the filtered image of the reconstructed image is saved in a decoded image cache as a reference image for other images; or, the filtered image of the reconstructed image is input into a display device for display, and is not used as a reference image.

14. An image decoding method, characterized in that: include: Decoding a code stream, determining a reconstructed image of a current image, and determining input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image; The input information is filtered by a neural network filter to obtain a filtered image of the reconstructed image; Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

15. An image encoding method, characterized in that: include: Encoding a current image, determining a reconstructed image of the current image, and determining input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image; The input information is filtered by a neural network filter to obtain a filtered image of the reconstructed image; Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

16. An image filtering device based on a neural network, characterized in that: include: A determination unit, configured to determine input information of a neural network filter based on the image to be filtered, wherein the input information includes the image to be filtered; A filtering unit, filtering the input information through a neural network filter to obtain a filtered image of the image to be filtered; Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

17. An image decoding device, characterized in that: include: A decoding unit, configured to decode a code stream, determine a reconstructed image of a current image, and determine input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image; A filtering unit, configured to filter the input information through a neural network filter to obtain a filtered image of the reconstructed image; Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

18. An image encoding device, characterized in that: include: An encoding unit, configured to encode a current image, determine a reconstructed image of the current image, and determine input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image; A filtering unit, configured to filter the input information through a neural network filter to obtain a filtered image of the reconstructed image; Among them, the neural network filter includes multiple convolutional layers for extracting image feature information, and the multiple convolutional layers are obtained by processing standard convolutional layers in at least two ways of a multi-level receptive field method, a tensor decomposition method and a grouped convolution method.

19. An electronic device comprising a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1 to 15.

20. A computer-readable storage medium, characterized in that: For storing computer programs; The computer program enables a computer to execute the method according to any one of claims 1 to 15.

Images