Image filtering method and apparatus based on neural network, and device and storage medium

Abstract

Provided in the embodiments of the present application are a filtering method and apparatus, a coding method and apparatus, a decoding method and apparatus, and an electronic device. The filtering method based on a neural network comprises: generating an input parameter for a neural network filter on the basis of an image to be subjected to filtering; inputting the input parameter into the neural network filter, wherein the neural network filter includes a residual unit used for extracting image feature information, the residual unit includes a plurality of residual blocks which are sequentially connected, each residual block includes a plurality of convolutional layers which are arranged in parallel and have convolution kernels of different sizes, and the residual unit comprises different first set convolutional layers and second set convolutional layers, which are alternately arranged; and acquiring a filtered image which is output by the neural network filter with regard to the image to be subjected to filtering. The technical solution in the embodiments of the present application can reduce the computational complexity of a neural network filter and improve the filtering effect, thereby facilitating an improvement in video coding and decoding efficiency.

Description

Image filtering method, device, equipment and storage medium based on neural network

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 28, 2023, with application number 2023109449208 and invention name “Filtering and encoding and decoding methods, devices and electronic equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image coding and decoding technology, and more specifically, to a filtering and coding and decoding method, device, and electronic device. Background Art

[0003] In the field of video encoding and decoding, a predicted image is superimposed with a reconstructed residual image to generate a reconstructed image. Because the reconstructed image will produce distortion, loop filtering is often required to obtain high-quality images. Improving the filtering effect during loop filtering to increase encoding and decoding efficiency is a pressing technical challenge.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a filtering and encoding and decoding method, device and electronic device, which can reduce the impact of asymmetry introduced by decomposition while reducing the computational complexity of the neural network filter, improve the filtering effect, and thus help improve the encoding and decoding efficiency of the video.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0007] According to one aspect of an embodiment of the present application, a neural network-based filtering method is provided, which is applied to a filtering device, and the method includes: generating input parameters of a neural network filter based on an image to be filtered; filtering the image to be filtered through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, the first set convolution layer is different from the second set convolution layer; and obtaining a filtered image output by the neural network filter for the image to be filtered.

[0008] According to one aspect of an embodiment of the present application, a video encoding method is provided, which is applied to an encoding device, and the method includes: generating input parameters of a neural network filter based on a reconstructed image to be filtered; processing the reconstructed image through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer arranged alternately, the first set convolution layer is different from the second set convolution layer; obtaining a filtered image output by the neural network filter for the reconstructed image; generating a predicted image corresponding to a next frame image based on the filtered image, and encoding the next frame of video image based on the predicted image corresponding to the next frame image.

[0009] According to one aspect of an embodiment of the present application, a video decoding method is provided, which is applied to a decoding device, the method comprising: generating input parameters of a neural network filter based on a reconstructed image to be filtered; filtering the reconstructed image through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer arranged alternately, the first set convolution layer is different from the second set convolution layer; obtaining a filtered image output by the neural network filter for the reconstructed image; generating a predicted image corresponding to a next frame image based on the filtered image, and decoding the video code stream based on the predicted image corresponding to the next frame image.

[0010] According to one aspect of an embodiment of the present application, a neural network-based filtering device is provided, which is applied to a filtering device, and the device includes: a generation unit, configured to generate input parameters of a neural network filter based on an image to be filtered; a processing unit, configured to filter the reconstructed image through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, the first set convolution layer is different from the second set convolution layer; an acquisition unit, configured to acquire a filtered image output by the neural network filter for the image to be filtered.

[0011] In some embodiments of the present application, based on the aforementioned scheme, at least one of the first set convolutional layer and the second set convolutional layer includes a convolutional layer obtained by decomposing by tensor decomposition, and the decomposition order of the first set convolutional layer and the second set convolutional layer is different.

[0012] In some embodiments of the present application, based on the aforementioned solution, the input parameters include at least one of the following information:

[0013] Boundary strength information, stripe-level quantization parameter, basic quantization parameter, frame type of the image to be filtered, and the predicted image corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

[0014] In some embodiments of the present application, based on the aforementioned scheme, the multiple convolutional layers include a first convolutional layer and a second convolutional layer, and the residual block also includes a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third convolutional layer and a fourth convolutional layer connected in sequence, and the output end of the first activation function layer and the output end of the second activation function layer are connected to the input end of the third convolutional layer; wherein the first convolutional layer and the second convolutional layer receive feature information of the input parameters, the feature information of the input parameters includes the image feature information, the first activation function layer and the second activation function layer are the same activation function layer or two independent activation function layers; the input parameters of the residual block are superimposed on the output data of the fourth convolutional layer and serve as the output data of the residual block.

[0015] In some embodiments of the present application, based on the aforementioned scheme, the two first convolution layers of two adjacent residual blocks are respectively the first setting convolution layer and the second setting convolution layer, and / or the two fourth convolution layers of two adjacent residual blocks are respectively the first setting convolution layer and the second setting convolution layer; or the first convolution layer of the same residual block is the first setting convolution layer, and the fourth convolution layer is the second setting convolution layer.

[0016] In some embodiments of the present application, based on the aforementioned scheme, the convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

[0017] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n by tensor decomposition; the convolution kernel size of the second convolutional layer is m×m, m and n are integers greater than or equal to 1 and m≠n.

[0018] In some embodiments of the present application, based on the aforementioned solution, the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n, and a convolutional layer with a convolution kernel size of n×1.

[0019] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution; the convolution kernel size of the second convolutional layer is m×m, and m and n are integers greater than or equal to 1 and m≠n.

[0020] In some embodiments of the present application, based on the aforementioned scheme, the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, and a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution.

[0021] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and depth-wise separable convolution; the convolution kernel size of the second convolutional layer is m×m, and m and n are integers greater than or equal to 1 and m≠n.

[0022] In some embodiments of the present application, based on the aforementioned scheme, the three sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution, and a convolutional layer with a convolution kernel size of 1×1.

[0023] In some embodiments of the present application, based on the above solution, the convolution kernel size of the fourth convolutional layer is k×k; or

[0024] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k by tensor decomposition; or

[0025] The fourth convolutional layer includes a subconvolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of k×k through group convolution; or

[0026] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through depthwise separable convolution; or

[0027] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through tensor decomposition and grouped convolution; or

[0028] The fourth convolutional layer includes three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through tensor decomposition and depth-wise separable convolution;

[0029] Here, k is an integer greater than or equal to 1.

[0030] In some embodiments of the present application, based on the aforementioned scheme, the neural network filter also includes: a shallow feature extraction unit, the shallow feature extraction unit includes at least one convolutional layer, and the shallow feature extraction unit is used to extract shallow feature information of the input parameters and input the shallow feature information into the residual unit.

[0031] In some embodiments of the present application, based on the aforementioned scheme, the input parameters include multiple information, the at least one convolutional layer includes multiple shallow feature extraction convolutional layers, each type of information corresponds to at least one shallow feature extraction convolutional layer, and the shallow feature extraction convolutional layer is used to extract shallow feature information of the corresponding information.

[0032] In some embodiments of the present application, based on the aforementioned scheme, the input parameters include multiple information, and the shallow feature extraction unit also includes a connection layer, which is used to connect the multiple information and input the connected multiple information into the at least one convolutional layer, and the at least one convolutional layer is used to extract shallow feature information of the connected multiple information and input the shallow feature information into the residual unit.

[0033] In some embodiments of the present application, based on the aforementioned solution, the residual unit is for both the image luminance component and the image chrominance component;

[0034] The neural network filter also includes: feature mapping units for image brightness components and image chrominance components, respectively, for mapping image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component and the feature mapping unit for the image chrominance component respectively process the output data of the residual unit.

[0035] In some embodiments of the present application, based on the aforementioned scheme, the residual unit includes residual units for image brightness components and image chrominance components respectively; the neural network filter also includes: feature mapping units for image brightness components and image chrominance components respectively, for mapping the image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component processes the output data of the residual unit for the image brightness component, and the feature mapping unit for the image chrominance component processes the output data of the residual unit for the image chrominance component.

[0036] According to one aspect of an embodiment of the present application, a video encoding device is provided, which is applied to an encoding device, and the device includes: a generation unit, configured to generate input parameters of a neural network filter based on a reconstructed image to be filtered; a processing unit, configured to process the reconstructed image through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, and the residual unit includes a first set convolution layer and a second set convolution layer arranged alternately, the first set convolution layer is different from the second set convolution layer; an acquisition unit, configured to obtain a filtered image output by the neural network filter for the reconstructed image; an encoding unit, configured to generate a predicted image corresponding to a next frame image based on the filtered image, and encode the next frame of video image based on the predicted image corresponding to the next frame image.

[0037] According to one aspect of an embodiment of the present application, a video decoding apparatus is provided, which is applied to a decoding device, and the method includes: a generating unit, configured to generate input parameters of a neural network filter based on a reconstructed image to be filtered; a processing unit, configured to filter the reconstructed image through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer arranged alternately, the first set convolution layer is different from the second set convolution layer; an acquiring unit, configured to acquire a filtered image output by the neural network filter for the reconstructed image; a decoding unit, configured to generate a predicted image corresponding to a next frame image based on the filtered image, and decode the video code stream based on the predicted image corresponding to the next frame image.

[0038] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the method described in the above embodiment.

[0039] According to one aspect of an embodiment of the present application, an encoder is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to perform the above encoding method.

[0040] According to one aspect of an embodiment of the present application, a decoder is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to perform the above-mentioned decoding method.

[0041] According to one aspect of an embodiment of the present application, a chip is provided for implementing the method in the above embodiment. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the method in the above embodiment.

[0042] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method in the above embodiment.

[0043] According to one aspect of an embodiment of the present application, a computer program product is provided, including computer program instructions, wherein the computer program instructions enable a computer to execute the method in the above embodiment.

[0044] According to one aspect of the embodiments of the present application, a computer program is provided, which, when executed on a computer, enables the computer to execute the method in the above embodiments.

[0045] In the technical solutions provided in some embodiments of the present application, since the neural network filter includes a residual unit, the residual unit includes a plurality of residual blocks connected in sequence, and the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the neural network filter can obtain feature information on multi-level receptive fields through the residual block, thereby improving the generalization ability of the neural network filter. The residual unit also includes different convolution layers arranged alternately, so that the impact of the asymmetry introduced by the decomposition can be reduced while reducing the computational complexity of the neural network filter, thereby improving the filtering effect, and thus helping to improve the encoding and decoding efficiency of the video.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] FIG1 is a schematic diagram showing an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied;

[0048] FIG2 is a schematic diagram showing the placement of a video encoding device and a video decoding device in a streaming transmission system;

[0049] FIG3 shows a basic flow chart of a video encoder;

[0050] FIG4 shows a schematic diagram of filtering processing based on NNLF;

[0051] FIG5 shows a schematic diagram of filtering the luminance component and the chrominance component;

[0052] FIG6 shows a schematic diagram of filtering the luminance component and the chrominance component;

[0053] FIG7 shows a flow chart of a filtering method based on a neural network according to an embodiment of the present application;

[0054] FIG8 shows a schematic diagram of a decomposition method according to CPD;

[0055] FIG9 shows a schematic diagram of an alternating decomposition method according to CPD;

[0056] FIG10 shows a schematic structural diagram of a residual block according to an embodiment of the present application;

[0057] FIG11 shows a schematic structural diagram of a residual block according to an embodiment of the present application;

[0058] FIG12 shows a flowchart of a video encoding method according to an embodiment of the present application;

[0059] FIG13 shows a flowchart of a video decoding method according to an embodiment of the present application;

[0060] FIG14 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0061] FIG15 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0062] FIG16 shows a schematic structural diagram of a residual block according to an embodiment of the present application;

[0063] FIG17 shows a schematic structural diagram of a residual block according to an embodiment of the present application;

[0064] FIG18 shows a schematic structural diagram of a residual block according to an embodiment of the present application;

[0065] FIG19 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0066] FIG20 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0067] FIG21 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0068] FIG22 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0069] FIG23 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0070] FIG24 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0071] FIG25 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0072] FIG26 shows a schematic structural diagram of a neural network filter according to an embodiment of the present application;

[0073] FIG27 shows a block diagram of a filtering device based on a neural network according to an embodiment of the present application;

[0074] FIG28 shows a block diagram of a video encoding apparatus according to an embodiment of the present application;

[0075] FIG29 shows a block diagram of a video decoding apparatus according to an embodiment of the present application;

[0076] FIG30 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0077] Example embodiments will now be described in a more complete manner with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to these examples; rather, these embodiments are provided to make this application more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art.

[0078] In addition, the features, structures or characteristics described in the present application may be combined in one or more embodiments in any suitable manner. In the following description, there are many specific details so that the embodiments of the present application can be fully understood. However, it will be appreciated by those skilled in the art that when implementing the technical solution of the present application, it is not necessary to use all the detailed features in the embodiments, one or more specific details may be omitted, or other methods, elements, devices, steps, etc. may be adopted.

[0079] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0080] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0081] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0082] FIG1 shows a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied.

[0083] 1 , a system architecture 100 includes a plurality of terminal devices, which can communicate with each other via, for example, a network 150. For example, the system architecture 100 may include a first terminal device 110 and a second terminal device 120 interconnected via the network 150. In the embodiment of FIG1 , the first terminal device 110 and the second terminal device 120 perform unidirectional data transmission.

[0084] For example, the first terminal device 110 can encode video data (such as a video picture stream captured by the terminal device 110) for transmission to the second terminal device 120 via the network 150. The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device 120 can receive the encoded video data from the network 150, decode the encoded video data to restore the video data, and display the video picture based on the restored video data.

[0085] In one embodiment of the present application, the system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 that perform bidirectional transmission of encoded video data, such as during a video conference. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 may encode video data (e.g., a video image stream captured by the terminal device) for transmission to the other of the third terminal device 130 and the fourth terminal device 140 via a network 150. Each of the third terminal device 130 and the fourth terminal device 140 may also receive the encoded video data transmitted by the other of the third terminal device 130 and the fourth terminal device 140, decode the encoded video data to recover the video data, and display the video image on an accessible display device based on the recovered video data.

[0086] In the embodiment of FIG1 , the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140 may be servers, personal computers, and smartphones, but the principles disclosed herein are not limited thereto. The embodiments disclosed herein are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks, including, for example, wired and / or wireless communication networks, for transmitting encoded video data between the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140. The communication network 150 may exchange data using circuit-switched and / or packet-switched channels. The network may include a telecommunications network, a local area network, a wide area network, and / or the Internet. For the purposes of this application, unless otherwise explained below, the architecture and topology of network 150 may be immaterial to the operations disclosed herein.

[0087] In one embodiment of the present application, FIG2 illustrates the placement of a video encoding device and a video decoding device in a streaming environment. The subject matter disclosed herein is equally applicable to other video-enabled applications, including, for example, video conferencing, digital television, and storing compressed video on digital media such as CDs, DVDs, and memory sticks.

[0088] The streaming system may include an acquisition subsystem 213, which may include a video source 201, such as a digital camera, that creates an uncompressed video picture stream 202. In one embodiment, the video picture stream 202 includes samples captured by the digital camera. The video picture stream 202 is depicted as a thicker line to emphasize the higher data volume of the video picture stream compared to the encoded video data 204 (or the encoded video stream 204). The video picture stream 202 may be processed by an electronic device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or embody various aspects of the disclosed subject matter, as described in greater detail below. The encoded video data 204 (or the encoded video stream 204) is depicted as a thinner line to emphasize the lower data volume of the encoded video data 204 (or the encoded video stream 204), which may be stored on a streaming server 205 for future use. One or more streaming client subsystems, such as client subsystem 206 and client subsystem 208 in FIG2 , can access streaming server 205 to retrieve copies 207 and 209 of encoded video data 204. Client subsystem 206 can include, for example, a video decoding device 210 in electronic device 230. Video decoding device 210 decodes the incoming copy 207 of the encoded video data and generates an output video picture stream 211 that can be presented on a display 212 (e.g., a screen) or another presentation device. In some streaming systems, the encoded video data 204, video data 207, and video data 209 (e.g., video code streams) can be encoded according to certain video encoding / compression standards.

[0089] It should be noted that the electronic device 220 and the electronic device 230 may include other components not shown in the figure. For example, the electronic device 220 may include a video decoding device, and the electronic device 230 may also include a video encoding device.

[0090] In one embodiment of the present application, taking the international video coding standards HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), and China's national video coding standard AVS as examples, after a video frame image is input, the video frame image will be divided into several non-overlapping processing units according to a block size, and each processing unit will perform similar compression operations. This processing unit is called CTU (Coding Tree Unit), or LCU (Largest Coding Unit). The CTU can be further divided into more refined parts to obtain one or more basic coding units CU (Coding Unit). CU is the most basic element in a coding link.

[0091] The following introduces some concepts when encoding CU:

[0092] Predictive Coding: Predictive coding includes intra-frame prediction and inter-frame prediction. The original video signal is predicted by a selected reconstructed video signal to produce a residual video signal. The encoder needs to decide which predictive coding mode to use for the current CU and inform the decoder. Intra-frame prediction means the predicted signal comes from an already coded and reconstructed area within the same image; inter-frame prediction means the predicted signal comes from a previously coded image (called a reference image) that is different from the current image.

[0093] Transform & Quantization: After the residual video signal undergoes transformations such as the Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT), the signal is converted to the transform domain, where the coefficients are known as transform coefficients. The transform coefficients are then subjected to a lossy quantization operation, which loses some information, making the quantized signal more suitable for compression. In some video coding standards, more than one transform scheme may be available, so the encoder must select one for the current CU and inform the decoder. The level of quantization is typically determined by the quantization parameter (QP). A larger QP value means that coefficients with a larger value range will be quantized to the same output, which generally results in greater distortion and a lower bitrate. Conversely, a smaller QP value means that coefficients with a smaller value range will be quantized to the same output, which generally results in less distortion and a higher bitrate.

[0094] Entropy Coding or Statistical Coding: The quantized transform domain signal is statistically compressed and encoded based on the frequency of occurrence of each value, and finally a binary (0 or 1) compressed code stream is output. At the same time, the encoding generates other information, such as the selected coding mode, motion vector data, etc., which also need to be entropy coded to reduce the bit rate. Statistical coding is a lossless coding method that can effectively reduce the bit rate required to express the same signal. Common statistical coding methods include variable length coding (VLC) or context-based binary arithmetic coding (CABAC).

[0095] The context-based binary arithmetic coding (CABAC) process consists of three main steps: binarization, context modeling, and binary arithmetic coding. After binarization of the input syntax elements, the binary data can be encoded using either the normal coding mode or the bypass coding mode. The bypass coding mode eliminates the need to assign a specific probability model to each binary bit. Instead, the input binary bit bin values ​​are directly encoded using a simple bypass encoder, speeding up both encoding and decoding. Generally, different syntax elements are not completely independent, and even the same syntax elements have some memory. Therefore, according to conditional entropy theory, conditional coding using other coded syntax elements can further improve coding performance compared to independent encoding or memoryless coding. This coded symbol information used as a condition is called context. In the normal coding mode, the binary bits of the syntax elements are sequentially fed into the context modeler. The encoder assigns an appropriate probability model to each input binary bit based on the values ​​of previously coded syntax elements or binary bits. This process is known as context modeling. The context model corresponding to the syntax element can be located using ctxIdxInc (context index increment) and ctxIdxStart (context index Start). After the bin value and the assigned probability model are fed into the binary arithmetic encoder for encoding, the context model needs to be updated based on the bin value, which is the adaptive process in encoding.

[0096] Loop Filtering: The changed and quantized signal will be reconstructed through inverse quantization, inverse transformation and prediction compensation operations to obtain a reconstructed image. Compared with the original image, due to the influence of quantization, some information of the reconstructed image is different from the original image, that is, the reconstructed image will produce distortion. Therefore, the reconstructed image can be filtered, such as deblocking filter (DB), SAO (Sample Adaptive Offset) or ALF (Adaptive Loop Filter) and other filters, which can effectively reduce the degree of distortion caused by quantization. Since these filtered reconstructed images will be used as a reference for subsequent encoded images to predict future image signals, the above filtering operation is also called loop filtering, that is, filtering operation within the encoding loop.

[0097] In one embodiment of the present application, FIG3 shows a basic flow chart of a video encoder, in which intra-frame prediction is used as an example for explanation. k [x,y] and predicted image signal Perform difference operation to obtain the residual signal u k [x,y], residual signal u k [x,y] is transformed and quantized to obtain the quantized coefficients. The quantized coefficients are entropy coded to obtain the encoded bit stream, and the reconstructed residual signal u' is obtained by inverse quantization and inverse transformation. k [x,y], predicted image signal and the reconstructed residual signal u' k [x,y] superposition generates reconstructed image signal Reconstructing image signals On the one hand, it is input to the intra-frame mode decision module and the intra-frame prediction module for intra-frame prediction processing, and on the other hand, it is filtered through loop filtering and outputs the filtered image signal s' k [x,y], filtered image signal s' k [x,y] can be used as the reference image for the next frame for motion estimation and motion compensation prediction. Then based on the result of motion compensation prediction s' r [x+m x ,y+m y ] and intra prediction results Get the predicted image signal of the next frame And continue to repeat the above process until the encoding is completed.

[0098] The above-mentioned loop filtering can be implemented based on NNLF (Neural Network Loop Filter, a loop filter based on a neural network). As shown in Figure 4, after the NNLF is trained, the image to be filtered can be input into the trained NNLF to obtain the filtered image. NNLF usually uses a loss function to constrain the filtered image so that it can be restored to the original image as much as possible. The loss function measures the difference between the predicted value and the true value. The greater the loss value between the predicted value and the true value, the greater the difference, and the goal of training is to reduce the loss value. Optionally, during training, NNLF can use L1 norm loss function and / or L2 norm loss function to construct the loss function of the model.

[0099] In order to flexibly process the luminance component and chrominance component of an image separately, a neural network-based loop filter (hereinafter referred to as a neural network filter) can also use a luminance and chroma separation (SLC) method.

[0100] For example, as shown in Figure 5, different network structures are used to construct neural network filters for the luminance component (Y) and the chrominance components (Cb and Cr). During training, different filter models can be trained separately for the luminance component (Y) and the chrominance components (Cb and Cr) to improve the filtering performance for the luminance component and the chrominance components. As shown in Figure 6, in the same network structure model, different modules in the network are used to process the luminance component and the chrominance component separately.

[0101] In video encoding and decoding based on neural networks, the neural network filters proposed in related technologies often build an overall network structure based on basic modules such as 3×3 convolution and residual blocks to train neural network filters. For the network structure of the neural network filter, repeatedly stacking 3×3 convolution and residual blocks can bring considerable coding performance, but it introduces excessively high computational complexity; limiting the number of repetitions of 3×3 convolution and residual blocks, that is, specifying a certain computational complexity, makes it difficult to obtain higher coding and decoding performance. Therefore, the technical solution of the embodiment of the present application proposes a new filtering solution based on a neural network, which can obtain higher coding performance without introducing too much complexity, thereby helping to improve the coding and decoding efficiency of the video.

[0102] It's important to note that neural networks fall under the umbrella of artificial intelligence (AI). AI is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also encompasses the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0103] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0104] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. The neural network filter in the embodiment of the present application is a filter based on machine learning / deep learning.

[0105] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:

[0106] FIG7 shows a flowchart of a neural network-based filtering method according to an embodiment of the present application. The neural network-based filtering method can be executed by a device with computing and processing capabilities, such as a terminal device or a server. For ease of description, the following description is based on an example of an execution subject being a filtering device. Referring to FIG7 , the neural network-based filtering method includes at least steps S710 to S730, which are described in detail as follows:

[0107] In step S710, the filtering device generates input parameters of the neural network filter based on the image to be filtered.

[0108] In one embodiment of the present application, the image to be filtered can be an image decoded by the video player, or an image captured. Alternatively, the image to be filtered can be a reconstructed image to be filtered, that is, an image generated by superimposing a reconstructed residual image obtained by inverse quantization and inverse transformation and a predicted image. For example, in the process shown in FIG3 , the reconstructed image is the predicted image signal. and the reconstructed residual signal u' k [x,y] Image signal generated by superposition The decoded reconstructed image to be filtered can be used for playback, or can be used for playback and provide a reference for prediction of subsequent images.

[0109] In some optional embodiments, the input parameters of the neural network filter may include at least one of the following information: boundary strength (i.e., boundary strength) information, slice-level quantization parameter (i.e., sliceQP), basic quantization parameter (i.e., baseQP), frame type of the image to be filtered (i.e., IPB), and the predicted image (i.e., prediction) corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

[0110] The frame type of the image to be filtered is used to indicate whether the reconstructed image is an I frame, a P frame, or a B frame. In the process shown in FIG3 , the predicted image corresponding to the reconstructed image is

[0111] In some optional embodiments, the image to be filtered may also be used as a type of information in the input parameters.

[0112] In step S720, based on the input parameters, the image to be filtered is filtered by a neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer arranged alternately, the first set convolution layer is different from the second set convolution layer, for example, the decomposition order of the first set convolution layer and the decomposition order of the second set convolution layer are different.

[0113] Optionally, the first set convolution layer and the second set convolution layer can be decomposed by tensor decomposition, and the tensor decomposition method can adopt CPD (Canonical Polyadic Decomposition). CPD is a classic tensor decomposition method that can be used to reduce the amount of convolution calculations in neural networks. As shown in Figure 8, based on the principle of CPD, a 3×3 convolution can be approximated by a 1×3 convolution and a 3×1 convolution, thereby effectively reducing the complexity of the convolution operation without significantly reducing network performance.

[0114] Optionally, the decomposition order of the first set convolution layer and the decomposition order of the second set convolution layer are different, that is, CPD decomposition can be used alternately. As shown in Figure 9, three 3×3 convolutions can be decomposed alternately according to the CPD principle to obtain a 1×3 convolution layer, a 3×1 convolution layer, a 3×1 convolution layer, a 1×3 convolution layer, a 1×3 convolution layer, and a 3×1 convolution layer. For example, the first set convolution layer can be a 1×3 convolution layer connected to a 3×1 convolution layer, and the second set convolution layer can be a 3×1 convolution layer connected to a 1×3 convolution layer; or, the first set convolution layer can be a 3×1 convolution layer connected to a 1×3 convolution layer, and the second set convolution layer can be a 1×3 convolution layer connected to a 3×1 convolution layer.

[0115] In an embodiment of the present application, since the residual block includes at least a plurality of convolutional layers arranged in parallel and with different convolution kernel sizes, the neural network filter can obtain feature information on multiple levels of receptive fields through the residual block, thereby improving the generalization ability of the neural network filter. Moreover, since the residual unit includes an alternating first set convolutional layer and a second set convolutional layer that can be decomposed by tensor decomposition, and the decomposition order of the first set convolutional layer and the second set convolutional layer is different, it is possible to reduce the influence of the asymmetry introduced by tensor decomposition while reducing the computational complexity of the neural network filter, thereby improving the filtering effect, and thereby helping to improve the encoding and decoding efficiency of the video.

[0116] It should be noted that in convolutional neural networks, the receptive field is the size of the region in the input feature map that maps a single element of each layer's output feature map. Different convolution window sizes (i.e., kernel sizes) process different numbers of input feature map elements, meaning that convolutional layers with different kernel sizes typically have different receptive fields.

[0117] In some optional embodiments, as shown in Figures 10 and 11, the multiple convolutional layers include a first convolutional layer and a second convolutional layer, and the residual block also includes a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third convolutional layer and a fourth convolutional layer connected in sequence; wherein, the input end of the first convolutional layer is connected to the input end of the second convolutional layer and serves as the input end of the residual block, the first convolutional layer and the second convolutional layer receive feature information of the input parameters, the feature information of the input parameters includes the image feature information, the first activation function layer and the second activation function layer are the same activation function layer or two independent activation function layers, and the output end of the first activation function layer and the output end of the second activation function layer are connected to the input end of the third convolutional layer; the input parameters of the residual block are superimposed with the output data of the fourth convolutional layer and serve as the output data of the residual block.

[0118] Optionally, as shown in FIG10 , the first activation function layer and the second activation function layer may be two independent activation function layers.

[0119] Optionally, as shown in FIG11 , the first activation function layer and the second activation function layer may be the same activation function layer.

[0120] It should be noted that the first activation function and the second activation function can be PReLU (Parametric Rectified Linear Unit), ReLU (Rectified Linear Unit), GeLU (Gaussian Error Linear Unit), etc., and the embodiments of the present application do not limit this.

[0121] In some optional embodiments, the two first convolutional layers of two adjacent residual blocks are respectively the first set convolutional layer and the second set convolutional layer, and / or the two fourth convolutional layers of two adjacent residual blocks are respectively the first set convolutional layer and the second set convolutional layer; or the first convolutional layer of the same residual block is the first set convolutional layer, and the fourth convolutional layer is the second set convolutional layer. That is, at least one of the first convolutional layer, the second convolutional layer, and the fourth convolutional layer of the embodiment of the present application includes a convolutional layer decomposed by tensor decomposition, and the decomposition order of the first set layer and the second convolutional layer is different. For example, in two adjacent residual blocks, the first convolutional layer of one residual block can be a 1×3 convolution connected to a 3×1 convolution, and the first convolutional layer of the other residual block can be a 3×1 convolution connected to a 1×3 convolution. For another example, the first convolutional layer of the same residual block can be a 1×3 convolution connected to a 3×1 convolution, and the fourth convolutional layer can be a 3×1 convolution connected to a 1×3 convolution.

[0122] In some optional embodiments, the convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n. For example, the convolution kernel size of the first convolution layer is 3×3, and the convolution kernel size of the second convolution layer is 1×1. In other embodiments of the present application, the values ​​of m and n can be, for example, 1, 3, 5, 7, etc.

[0123] In some optional embodiments, the first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n1×n2 by tensor decomposition; the convolution kernel size of the second convolutional layer is m1×m2, and m1, m2, n1 and n2 are integers greater than or equal to 1.

[0124] Optionally, the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n1, and a convolutional layer with a convolution kernel size of n2×1.

[0125] Optionally, n1≠n2, for example, the first convolutional layer includes a 3×1 sub-convolutional layer and a 1×2 sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of 3×2 by tensor decomposition.

[0126] Optionally, n1=n2. For example, the first convolutional layer includes a 3×1 sub-convolutional layer and a 1×3 sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of 3×3 by tensor decomposition.

[0127] In some optional embodiments, the first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution; the convolution kernel size of the second convolutional layer is m×m, where m and n are integers greater than or equal to 1 and m≠n. For example, the first convolutional layer includes a 3×1 sub-convolutional layer and a 1×3 sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of 3×3, and the convolution kernel size of the second convolutional layer is 1×1. In other embodiments of the present application, the values ​​of m and n can be, for example, 1, 3, 5, 7, etc.

[0128] Optionally, the two sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution include: the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and grouped convolution, and a convolutional layer with a convolution kernel size of n×1 and grouped convolution.

[0129] In some optional embodiments, the first convolutional layer includes: three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and depthwise separable convolution; the convolution kernel size of the second convolutional layer is m×m, where m and n are integers greater than or equal to 1 and m≠n. For example, the first convolutional layer includes a 3×1 sub-convolutional layer and a 1×3 sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of 3×3, and the convolution kernel size of the second convolutional layer is 1×1. In other embodiments of the present application, the values ​​of m and n can be, for example, 1, 3, 5, 7, etc.

[0130] Among them, Depth-wise Separable Convolution (DSC) is an improved algorithm of standard convolution. It first convolves each channel separately with a given convolution kernel size and combines the results. This part is called depth-wise convolution. Then, the depth-wise separable convolution uses a 1×1 convolution kernel for ordinary / standard convolution and outputs a feature map. This part is called pointwise convolution. Depth-wise separable convolution DSC can effectively reduce the complexity of convolution operations.

[0131] Optionally, the convolution layer with a convolution kernel size of n×n can be first decomposed by depthwise separable convolution to obtain a convolution layer with a convolution kernel size of n×n and grouped convolution, and a convolution layer with a convolution kernel size of 1×1; and then the convolution layer with a convolution kernel size of n×n and grouped convolution can be decomposed by tensor decomposition into a convolution layer with a convolution kernel size of 1×n and grouped convolution, and a convolution layer with a convolution kernel size of n×1 and grouped convolution.

[0132] In some optional embodiments, the fourth convolutional layer may be a convolutional layer with a convolution kernel size of k×k, where k is an integer greater than or equal to 1, such as 1, 3, 5, 7, etc. Or the fourth convolutional layer may also be a plurality of sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k by tensor decomposition, and the convolution kernel sizes of the two sub-convolutional layers may be 1×k and k×1 respectively; or the fourth convolutional layer may include a sub-convolutional layer obtained by decomposing the convolutional layer with a convolution kernel size of k×k by grouped convolution; or the fourth convolutional layer may include two sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k by depthwise separable convolution, and the two sub-convolutional layers may be a convolutional layer with a convolution kernel size of k×k and grouped convolution, and a convolutional layer with a convolution kernel size of 1×1; or the fourth The convolution layer includes two sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of k×k through tensor decomposition and grouped convolution, and the two sub-convolution layers can be a convolution layer with a convolution kernel size of 1×k and grouped convolution, and a convolution layer with a convolution kernel size of k×1 and grouped convolution; or the fourth convolution layer includes three sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of k×k through tensor decomposition and depthwise separable convolution, and the three sub-convolution layers can be a convolution layer with a convolution kernel size of 1×k and grouped convolution, a convolution layer with a convolution kernel size of k×1 and grouped convolution, and a convolution layer with a convolution kernel size of 1×1.

[0133] In some optional embodiments, the residual unit in the neural network filter may be luminance- and chrominance-separated, that is, the residual unit in the neural network filter includes residual units for the image luminance component and the image chrominance component, respectively. In this case, the residual block structure included in the residual unit for the image luminance component may be the same as or different from the residual block structure included in the residual unit for the image chrominance component.

[0134] In some optional embodiments, the neural network filter may further include: a shallow feature extraction unit, the shallow feature extraction unit includes at least one convolutional layer, the shallow feature extraction unit is used to extract shallow feature information of the input parameters and input the shallow feature information into the residual unit.

[0135] Optionally, at least one convolutional layer of the shallow feature extraction unit includes a convolutional layer with a convolution kernel size of r×r, where r is an integer greater than or equal to 1, such as 1, 3, 5, 7, etc. The at least one convolutional layer of the shallow feature extraction unit may also be a plurality of sub-convolutional layers obtained by decomposition through tensor decomposition and / or depthwise separable convolution or grouped convolution. For example, at least one convolutional layer of the shallow feature extraction unit includes a convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of r×r through grouped convolution; or, at least one convolutional layer of the shallow feature extraction unit includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of r×r through tensor decomposition or depth-wise separable convolution; or, at least one convolutional layer of the shallow feature extraction unit includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of r×r through tensor decomposition and grouped convolution; or, at least one convolutional layer of the shallow feature extraction unit includes three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of r×r through tensor decomposition and depth-wise separable convolution.

[0136] In some optional embodiments, the input parameters include multiple information, at least one convolutional layer includes multiple shallow feature extraction convolutional layers, each type of information corresponds to at least one shallow feature extraction convolutional layer, and the shallow feature extraction convolutional layer is used to extract shallow feature information of the corresponding information.

[0137] In some optional embodiments, the input parameters include multiple information, and the shallow feature extraction unit further includes a connection layer, which is used to connect the multiple information and input the connected multiple information into at least one convolutional layer, and the at least one convolutional layer is used to extract shallow feature information of the connected multiple information and input the shallow feature information into the residual unit.

[0138] In some optional embodiments, the neural network filter may further include: a feature mapping unit for mapping image feature information output by the residual unit. Optionally, if the residual unit in the neural network filter targets both the image luminance component and the image chrominance component, then feature mapping units for the image luminance component and the image chrominance component may be provided separately; wherein the feature mapping unit for the image luminance component and the feature mapping unit for the image chrominance component are respectively connected to the residual unit, and the feature mapping unit for the image luminance component and the feature mapping unit for the image chrominance component respectively process the output data of the residual unit.

[0139] Optionally, the feature mapping unit includes a convolution layer with a convolution kernel size of s×s, where s is an integer greater than or equal to 1, such as 1, 3, 5, 7, etc. The convolution layer included in the feature mapping unit may also be a plurality of sub-convolution layers obtained by decomposition through tensor decomposition and / or depthwise separable convolution or grouped convolution. For example, for a feature mapping unit, it includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using tensor decomposition; or, for a feature mapping unit, it includes one sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using grouped convolution; or, for a feature mapping unit, it includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using depthwise separable convolution; or, for a feature mapping unit, it includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using tensor decomposition and grouped convolution; and three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using tensor decomposition and depthwise separable convolution.

[0140] In some optional embodiments, if the residual unit in the neural network filter includes residual units for the image brightness component and the image chrominance component respectively, then feature mapping units for the image brightness component and the image chrominance component can be set respectively, and the feature mapping unit for the image brightness component is connected to the residual unit for the image brightness component, and the feature mapping unit for the image brightness component processes the output data of the residual unit for the image brightness component, and the feature mapping unit for the image chrominance component is connected to the residual unit for the image chrominance component, and the feature mapping unit for the image chrominance component processes the output data of the residual unit for the image chrominance component.

[0141] In step S730, a filtered image output by the neural network filter for the reconstructed image is obtained.

[0142] In one embodiment of the present application, the neural network filter needs to use the same parameters as those used in application as input during training. Specifically, during the training phase, input parameters for training the neural network filter are generated based on the sample image (the input parameters are adjusted according to the usage scenario of the neural network filter, that is, the same as the parameters used when the neural network filter is in use), and then the obtained input parameters are input into the neural network filter. The parameters of the neural network filter are adjusted according to the loss value between the output of the neural network filter and the expected filtering result image corresponding to the sample image, and this process is repeated until the neural network filter meets the convergence condition.

[0143] It should be noted that the technical solution of the embodiment shown in FIG7 can be applied to the loop filter in the video encoding and decoding process, that is, loop filtering is performed in the video encoding and decoding process using the filtering method shown in FIG7. Alternatively, the technical solution of the embodiment shown in FIG7 can also be applied to post-processing of videos or images, that is, filtering images decoded by the video playback end, images captured by the terminal device, or other images.

[0144] Based on the neural network-based filtering method shown in FIG7 , the present application also proposes a video encoding method in an embodiment. The video encoding method can be performed by a device with computing and processing capabilities, such as an encoding device. The specific process is shown in FIG12 , including the following steps S1210 to S1240:

[0145] In step S1210, input parameters of the neural network filter are generated based on the reconstructed image to be filtered.

[0146] The specific implementation details of this step can refer to the aforementioned step S710 and will not be repeated here.

[0147] In step S1220, based on the input parameters, the image is reconstructed and filtered through a neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, and the residual unit includes a first set convolution layer and a second set convolution layer arranged alternately, and the first set convolution layer is different from the second set convolution layer.

[0148] The specific implementation details of this step can be referred to the aforementioned step S720 and will not be repeated here.

[0149] In step S1230, a filtered image output by the neural network filter for the reconstructed image is obtained.

[0150] The specific implementation details of this step can refer to the aforementioned step S730 and will not be repeated here.

[0151] In step S1240 , a predicted image corresponding to the next frame of image is generated based on the filtered image, and encoding processing is performed on the next frame of video image based on the predicted image corresponding to the next frame of image.

[0152] Optionally, after obtaining the filtered image, the process shown in Figure 3 can be referred to, that is, the filtered image of the reconstructed image is used as the reference image of the next frame image for motion estimation and motion compensation prediction, and then the predicted image of the next frame image is obtained based on the result of the motion compensation prediction and the intra-frame prediction result, and the process shown in Figure 3 is repeated until the video image encoding is completed.

[0153] Accordingly, based on the neural network-based filtering method shown in FIG7 , the present application also proposes a video decoding method in an embodiment. The video decoding method can be performed by a device with computing and processing capabilities, such as a decoding device. The specific process is shown in FIG13 , and includes the following steps S1310 to S1340:

[0154] In step S1310, input parameters of the neural network filter are generated based on the reconstructed image to be filtered.

[0155] The specific implementation details of this step can refer to the aforementioned step S710 and will not be repeated here.

[0156] In step S1320, the reconstructed image is filtered by a neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, and the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, and the first set convolution layer is different from the second set convolution layer.

[0157] The specific implementation details of this step can be referred to the aforementioned step S720 and will not be repeated here.

[0158] In step S1330, a filtered image output by the neural network filter for the reconstructed image is obtained.

[0159] The specific implementation details of this step can refer to the aforementioned step S730 and will not be repeated here.

[0160] In step S1340 , a predicted image corresponding to the next frame image is generated based on the filtered image, and the video stream is decoded based on the predicted image corresponding to the next frame image.

[0161] Optionally, after obtaining the filtered image of the reconstructed image, the filtered image of the reconstructed image can be used as the reference image of the next frame for motion estimation and motion compensation prediction, and then the predicted image of the next frame is obtained based on the result of the motion compensation prediction and the intra-frame prediction result. The predicted image of the next frame is superimposed again with the reconstructed residual signal obtained by inverse quantization and inverse transformation to generate the reconstructed image of the next frame, and this process is repeated to realize the decoding processing of the video code stream.

[0162] The following is a further description of the implementation details not detailed in the embodiments of the present application with reference to FIG. 14 to FIG. 26 :

[0163] In the embodiments of this application, a neural network-based filtering solution is proposed. This solution utilizes methods such as luminance and chrominance separation, multi-level receptive fields, and alternating CPD to design a novel neural network filter structure, thereby achieving a good balance between encoding performance and computational complexity. This solution can be applied to video codecs or video pre- and post-processing products.

[0164] In some optional embodiments, as shown in FIG14 and FIG15 , a structure of a neural network filter in an embodiment of the present application is shown, which includes an input part, a head part, a middle part, a tail part and an output part.

[0165] Optionally, the input part of the neural network filter can include two parts: image information and side information. Among them, the image information includes the reconstructed image to be filtered, the predicted image corresponding to the reconstructed image to be filtered, and the boundary strength, so as to provide the main image content for the filtering of the reconstructed image for encoding and decoding. The side information includes sliceQP, baseQP and IPB. Among them, sliceQP and baseQP are used to indicate the quantization parameter and distortion level of the reconstructed image to be filtered, and IPB is used to indicate the encoding type of the reconstructed image to be filtered (i.e., which one of I frame, P frame and B frame).

[0166] It should be noted that the input part of the neural network filter includes but is not limited to the above information. In addition, the reconstructed image to be filtered is essential in the input part of the neural network filter, and other information can be deleted or added according to actual needs.

[0167] In some optional embodiments, the neural network filter head is used to extract shallow feature information of the input parameters and may include two 3×3 convolutional layers and a PReLU activation layer. The input parameters are concatenated and sent to the neural network filter head module for feature extraction to obtain shallow feature representations.

[0168] In some optional embodiments, the middle part of the neural network filter is an important backbone structure of the network and may include multiple residual blocks (ResBlocks). Optionally, the middle part may adopt the form of luminance and chrominance separation, that is, using different modules Y-Part and UV-Part to separately process the luminance component and chrominance component.

[0169] In one embodiment of the present application, the structure of the residual block can be as shown in Figure 16. According to the CPD principle, a 3×3 convolution is decomposed into a 3×1 convolution and a 1×3 convolution, and the decomposition is performed alternately within the same residual block, so that the first convolution layer in the same residual block is a 1×3 convolution and a 3×1 convolution, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution.

[0170] In one embodiment of the present application, the structure of the residual block can be as shown in Figure 17, where CPD and GC are used simultaneously to decompose a 3×3 convolution into a 1×3 convolution for grouped convolution and a 3×1 convolution. Wherein, g indicates the use of grouped convolution, that is, the convolution within the specified channel range uses the same parameters and is alternately decomposed within the same residual block, so that the first convolution layer in the same residual block is a 1×3 convolution and a 3×1 convolution for grouped convolution, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution.

[0171] In one embodiment of the present application, the structure of the residual block can be as shown in Figure 18, using CPD and DSC at the same time to divide a 3×3 convolution into a 1×3 convolution with grouped convolution, a 3×1 convolution with grouped convolution, and a 1×1 convolution. Where g indicates the use of grouped convolution, that is, the convolution within the specified channel range uses the same parameters.

[0172] It should be noted that each residual block contained in the Y-Part and UV-Part in the middle part shown in Figure 15 can adopt any of the structures shown in Figures 16 to 18, or some of the residual blocks therein can adopt any of the structures shown in Figures 16 to 18. For example, if the Y-Part contains four residual blocks, all four of them can adopt the structure shown in Figure 16; or two of them can adopt the structure shown in Figure 17, and the other two can adopt the structure shown in Figure 18.

[0173] Optionally, the number and / or structure of residual blocks included in the Y-Part and UV-Part can be the same or different. For example, different residual blocks can be used in the Y-Part and UV-Part in the middle portion. For example, the Y-Part can use one or more residual blocks shown in Figures 16 to 18, while the UV-Part uses another residual block.

[0174] In some optional embodiments, as shown in FIG15 , when the brightness and chromaticity in the middle portion are not separated, the same tail structure can be used to obtain a filtered image of brightness and a filtered image of chromaticity; when the brightness and chromaticity in the middle portion are separated, the tail network of the neural network filter can also adopt the form of brightness and chromaticity separation to obtain a filtered image of brightness and a filtered image of chromaticity respectively. Optionally, the tail can include two convolutional layers and a PReLU activation layer to map feature information with a larger number of channels into image information with a smaller number of channels.

[0175] In some optional embodiments, the reconstructed image to be filtered input by the neural network filter and the residual obtained by network learning are added to obtain the final output of the network, which is the filtered image. That is, in the neural network filter structure shown in Figure 15, the neural network is used to learn the residual data. In this way, since the amount of residual data is small, the processing efficiency of the neural network can be accelerated. In other embodiments of the present application, the output of the tail network of the neural network filter can also be directly used as the output of the neural network filter, so that the neural network filter learns the difference between the image before and after filtering.

[0176] It should be noted that the structure of the residual block and the size of the convolution kernel in the embodiment of the present application are only examples. In other embodiments of the present application, they can be adjusted according to the actual application scenario.

[0177] In some optional embodiments, as shown in Figures 19 to 26, the head of the neural network filter includes multiple 3×3 convolutional layers and multiple PReLU activation layers corresponding to different inputs, and includes a 1×1 convolutional layer and a PReLU activation layer, and multiple PReLU activation layers corresponding to different inputs are jointly connected to the 1×1 convolutional layer.

[0178] In one embodiment of the present application, the structure of the neural network filter is shown in Figure 19. The middle part of the neural network filter is composed of a plurality of first residual blocks and a plurality of second residual blocks arranged alternately, and the tail of the neural network filter is a single-branch structure. ResBlockT1 represents the first residual block, ResBlockT2 represents the second residual block, the first convolution layer of ResBlockT1 is to decompose a 3×3 convolution into a 1×3 convolution and a 3×1 convolution according to the CPD principle, and the first convolution layer of ResBlockT2 is to decompose a 3×3 convolution into a 3×1 convolution and a 1×3 convolution according to the CPD principle. The tail contains 2 3×3 convolution layers and a PReLU activation layer.

[0179] In one embodiment of the present application, the structure of the neural network filter is shown in Figure 20. The middle part of the neural network filter is composed of a plurality of first residual blocks and second residual blocks that are decomposed according to CPD and GC and arranged alternately, and the tail of the neural network filter is a single-branch structure. ResBlockT1 represents the first residual block, ResBlockT2 represents the second residual block, the first convolution layer of ResBlockT1 is a 1×3 convolution and a 3×1 convolution obtained by decomposing a 3×3 convolution using CPD and GC, and the first convolution layer of ResBlockT2 is a 3×1 convolution and a 1×3 convolution obtained by decomposing a 3×3 convolution using CPD and GC; the tail contains 2 3×3 convolution layers and a PReLU activation layer.

[0180] In one embodiment of the present application, the structure of the neural network filter is shown in Figure 21. The middle part of the neural network filter is composed of a plurality of residual blocks that are alternately decomposed according to the CPD principle. ResBlockT3 represents the residual block, that is, the first convolution layer of ResBlockT3 is a 1×3 convolution and a 3×1 convolution, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution; and the tail of the neural network filter is a single-branch structure, including two 3×3 convolution layers and a PReLU activation layer.

[0181] In one embodiment of the present application, the structure of the neural network filter is shown in Figure 22. The middle part of the neural network filter is composed of a plurality of residual blocks obtained by alternating decomposition according to CPD and GC, and ResBlockT3 represents the residual block, that is, the first convolution layer of ResBlockT3 is a 1×3 convolution and a 3×1 convolution obtained by decomposing a 3×3 convolution using CPD and GC, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution obtained by decomposing a 3×3 convolution using CPD and GC; and the tail of the neural network filter is a single-branch structure, including two 3×3 convolution layers and a PReLU activation layer.

[0182] In one embodiment of the present application, the structure of the neural network filter is shown in Figure 23. The middle part of the neural network filter is composed of a plurality of first residual blocks and a plurality of second residual blocks arranged alternately, and the tail of the neural network filter is composed of two branches. Among them, ResBlockT1 represents the first residual block, ResBlockT2 represents the second residual block, the first convolution layer of ResBlockT1 is to decompose a 3×3 convolution into a 1×3 convolution and a 3×1 convolution according to the CPD principle, and the first convolution layer of ResBlockT2 is to decompose a 3×3 convolution into a 3×1 convolution and a 1×3 convolution according to the CPD principle; the two branches at the tail include a luminance branch for the luminance component of the image and a chrominance branch for the chrominance component of the image. The luminance branch and the chrominance branch respectively contain 2 3×3 convolution layers and a PReLU activation layer.

[0183] In one embodiment of the present application, the structure of the neural network filter is shown in Figure 24. The middle part of the neural network filter is composed of a plurality of first residual blocks and second residual blocks that are decomposed according to CPD and GC and arranged alternately, and the tail of the neural network filter is a two-branch structure. Among them, ResBlockT1 represents the first residual block, ResBlockT2 represents the second residual block, the first convolution layer of ResBlockT1 is a 1×3 convolution and a 3×1 convolution obtained by decomposing a 3×3 convolution using CPD and GC, and the first convolution layer of ResBlockT2 is a 3×1 convolution and a 1×3 convolution obtained by decomposing a 3×3 convolution using CPD and GC; the two branches at the tail include a luminance branch for the luminance component of the image and a chrominance branch for the chrominance component of the image. The luminance branch and the chrominance branch each contain two 3×3 convolution layers and a PReLU activation layer.

[0184] In one embodiment of the present application, the structure of the neural network filter is shown in Figure 25. The middle part of the neural network filter is composed of a plurality of residual blocks that are alternately decomposed according to the CPD principle. ResBlockT3 represents the residual block, that is, the first convolution layer of ResBlockT3 is a 1×3 convolution and a 3×1 convolution, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution; and the tail of the neural network filter includes two branches, the two branches of the tail include a luminance branch for the luminance component of the image and a chrominance branch for the chrominance component of the image, and the luminance branch and the chrominance branch respectively include two 3×3 convolution layers and a PReLU activation layer.

[0185] In one embodiment of the present application, the structure of the neural network filter is shown in Figure 26. The middle part of the neural network filter is composed of a plurality of third residual blocks obtained by alternating decomposition and GC according to CPD, and ResBlockT3 represents the residual block, that is, the first convolution layer of ResBlockT3 is a 1×3 convolution and a 3×1 convolution obtained by decomposing a 3×3 convolution using CPD and GC, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution obtained by decomposing a 3×3 convolution using CPD and GC; and the tail of the neural network filter includes two branches, the two branches of the tail include a luminance branch for the luminance component of the image and a chrominance branch for the chrominance component of the image, and the luminance branch and the chrominance branch respectively include two 3×3 convolution layers and a PReLU activation layer.

[0186] The following describes an embodiment of the device of the present application, which can be used to perform the method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method in the above embodiment of the present application.

[0187] FIG18 shows a block diagram of a filtering device based on a neural network according to an embodiment of the present application. The filtering device based on a neural network can be applied to a device with a computing and processing function, such as a filtering device.

[0188] 27 , a neural network-based filtering device 2700 according to an embodiment of the present application includes: a generating unit 2702 , a processing unit 2704 , and an acquiring unit 2706 .

[0189] Among them, the generation unit 2702 is configured to generate input parameters of the neural network filter based on the image to be filtered; the processing unit 2704 is configured to filter the image to be filtered through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, and the first set convolution layer is different from the second set convolution layer; the acquisition unit 2706 is configured to obtain the filtered image output by the neural network filter for the image to be filtered.

[0190] In some embodiments of the present application, based on the aforementioned scheme, at least one of the first set convolutional layer and the second set convolutional layer includes a convolutional layer obtained by decomposing by tensor decomposition, and the decomposition order of the first set convolutional layer and the second set convolutional layer is different.

[0191] In some embodiments of the present application, based on the aforementioned solution, the input parameters include at least one of the following information:

[0192] Boundary strength information, stripe-level quantization parameter, basic quantization parameter, frame type of the image to be filtered, and the predicted image corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

[0193] In some embodiments of the present application, based on the aforementioned scheme, multiple convolutional layers include a first convolutional layer and a second convolutional layer, and the residual block also includes a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third convolutional layer and a fourth convolutional layer connected in sequence, and the output end of the first activation function layer and the output end of the second activation function layer are connected to the input end of the third convolutional layer; wherein the first convolutional layer and the second convolutional layer receive feature information of the input parameters, the feature information of the input parameters includes the image feature information, the first activation function layer and the second activation function layer are the same activation function layer or two independent activation function layers; the input parameters of the residual block are superimposed on the output data of the fourth convolutional layer and serve as the output data of the residual block.

[0194] In some embodiments of the present application, based on the aforementioned scheme, the two first convolution layers of two adjacent residual blocks are respectively the first set convolution layer and the second set convolution layer, and / or the two fourth convolution layers of two adjacent residual blocks are respectively the first set convolution layer and the second set convolution layer; or the first convolution layer of the same residual block is the first set convolution layer, and the fourth convolution layer is the second set convolution layer.

[0195] In some embodiments of the present application, based on the aforementioned scheme, the convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

[0196] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution; the convolution kernel size of the second convolutional layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

[0197] In some embodiments of the present application, based on the aforementioned scheme, the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, and a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution.

[0198] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and depth-wise separable convolution; the convolution kernel size of the second convolutional layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

[0199] In some embodiments of the present application, based on the aforementioned scheme, the three sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution, and a convolutional layer with a convolution kernel size of 1×1.

[0200] In some embodiments of the present application, based on the above solution, the convolution kernel size of the fourth convolutional layer is k×k; or

[0201] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k by tensor decomposition; or

[0202] The fourth convolutional layer includes a subconvolutional layer obtained by decomposing the convolutional layer with a convolution kernel size of k×k through grouped convolution; or

[0203] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k through depthwise separable convolution; or

[0204] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k through tensor decomposition and grouped convolution; or

[0205] The fourth convolutional layer includes three sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k through tensor decomposition and depth-wise separable convolution;

[0206] Here, k is an integer greater than or equal to 1.

[0207] In some embodiments of the present application, based on the aforementioned scheme, the residual unit includes residual units for the image luminance component and the image chrominance component respectively; wherein the residual block structure contained in the residual unit for the image luminance component is the same as or different from the residual block structure contained in the residual unit for the image chrominance component.

[0208] In some embodiments of the present application, based on the aforementioned scheme, the neural network filter also includes: a shallow feature extraction unit, the shallow feature extraction unit includes at least one convolutional layer, the shallow feature extraction unit is used to extract shallow feature information of the input parameters and input the shallow feature information into the residual unit.

[0209] In some embodiments of the present application, based on the aforementioned scheme, the input parameters include multiple information, at least one convolutional layer includes multiple shallow feature extraction convolutional layers, each type of information corresponds to at least one shallow feature extraction convolutional layer, and the shallow feature extraction convolutional layer is used to extract shallow feature information of the corresponding information.

[0210] In some embodiments of the present application, based on the aforementioned scheme, the input parameters include multiple information, and the shallow feature extraction unit also includes a connection layer, which is used to connect the multiple information and input the connected multiple information into at least one convolutional layer. The at least one convolutional layer is used to extract shallow feature information of the connected multiple information and input the shallow feature information into the residual unit.

[0211] In some embodiments of the present application, based on the aforementioned solution, the residual unit is directed to both the image luminance component and the image chrominance component;

[0212] The neural network filter also includes: feature mapping units for image brightness components and image chrominance components, respectively, for mapping image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component and the feature mapping unit for the image chrominance component respectively process the output data of the residual unit.

[0213] In some embodiments of the present application, based on the aforementioned scheme, the residual unit includes residual units for the image brightness component and the image chrominance component respectively; the neural network filter also includes: feature mapping units for the image brightness component and the image chrominance component respectively, for mapping the image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component processes the output data of the residual unit for the image brightness component, and the feature mapping unit for the image chrominance component processes the output data of the residual unit for the image chrominance component.

[0214] FIG28 shows a block diagram of a video encoding apparatus according to an embodiment of the present application, which can be applied to a device having a computing and processing function, such as an encoding device.

[0215] 28 , a video encoding apparatus 2800 according to an embodiment of the present application includes: a generating unit 2802 , a processing unit 2804 , an acquiring unit 2806 , and an encoding unit 2808 .

[0216] Among them, the generation unit 2802 is configured to generate input parameters of the neural network filter based on the reconstructed image to be filtered; the processing unit 2804 is configured to filter the reconstructed image through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, and the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, and the first set convolution layer is different from the second set convolution layer; the acquisition unit 2806 is configured to obtain a filtered image for the reconstructed image output by the neural network filter; the encoding unit 2808 is configured to generate a predicted image corresponding to the next frame image based on the filtered image, and encode the next frame of video image based on the predicted image corresponding to the next frame image.

[0217] FIG20 shows a block diagram of a video decoding apparatus according to an embodiment of the present application. The video decoding apparatus can be applied to a device having a computing and processing function, such as a decoding device.

[0218] 29 , a video decoding apparatus 2900 according to an embodiment of the present application includes: a generating unit 2902 , a processing unit 2904 , an acquiring unit 2906 , and a decoding unit 2908 .

[0219] Among them, the generation unit 2902 is configured to generate input parameters of the neural network filter based on the reconstructed image to be filtered; the processing unit 2904 is configured to filter the reconstructed image through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer arranged alternately, and the first set convolution layer is different from the second set convolution layer; the acquisition unit 2906 is configured to obtain a filtered image for the reconstructed image output by the neural network filter; the decoding unit 2908 is configured to generate a predicted image corresponding to the next frame image based on the filtered image, and decode the video code stream based on the predicted image corresponding to the next frame image.

[0220] FIG30 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.

[0221] It should be noted that the computer system 3000 of the electronic device shown in FIG30 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0222] As shown in Figure 30, computer system 3000 may include a central processing unit (CPU) 3001, which can perform various appropriate actions and processes according to the program stored in read-only memory (ROM) 3002 or the program loaded from storage portion 3008 into random access memory (RAM) 3003, such as the method for executing the above embodiment. Various programs and data required for system operation are also stored in RAM 3003. CPU 3001, ROM 3002 and RAM 3003 are connected to each other via bus 3004. Input / output (I / O) interface 3005 is also connected to bus 3004.

[0223] The following components can be connected to the I / O interface 3005: an input section 3006 including a keyboard, a mouse, and the like; an output section 3007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 3008 including a hard disk and the like; and a communication section 3009 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 3009 performs communication processing via a network such as the Internet. A drive 3010 is also connected to the I / O interface 3005 as needed. Removable media 3011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 3010 as needed, so that computer programs read from the removable media can be installed in the storage section 3008 as needed.

[0224] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program is used to perform the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 3009, and / or installed from a removable medium 3011. When the computer program is executed by the central processing unit (CPU) 3001, the various functions defined in the system of the present application are performed.

[0225] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a computer program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0226] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0227] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0228] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more computer programs, and when the one or more computer programs are executed by the electronic device, the electronic device implements the method described in the above embodiments.

[0229] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed by a computer, enables the computer to perform the method of the above method embodiment.

[0230] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0231] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable an electronic device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0232] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0233] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A filtering method based on a neural network, characterized in that: include: Based on the image to be filtered, generating input parameters of the neural network filter; Based on the input parameters, the image to be filtered is filtered by the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; Obtain a filtered image output by the neural network filter for the image to be filtered.

2. The filtering method based on a neural network according to claim 1, characterized in that: At least one of the first set convolutional layer and the second set convolutional layer includes a convolutional layer decomposed by tensor decomposition, and the decomposition order of the first set convolutional layer and the second set convolutional layer is different.

3. The neural network-based filtering method according to claim 1 or 2, characterized in that: The input parameters include at least one of the following information: Boundary strength information, strip-level quantization parameter, basic quantization parameter, frame type of the image to be filtered, and the predicted image corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

4. The filtering method based on a neural network according to any one of claims 1 to 3, characterized in that: The multiple convolutional layers include a first convolutional layer and a second convolutional layer, the residual block also includes a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third convolutional layer and a fourth convolutional layer connected in sequence, and the output end of the first activation function layer and the output end of the second activation function layer are connected to the input end of the third convolutional layer; Among them, the first convolution layer and the second convolution layer receive feature information of the input parameters, the feature information of the input parameters includes the image feature information, the first activation function layer and the second activation function layer are the same activation function layer or two independent activation function layers; the input parameters of the residual block are superimposed with the output data of the fourth convolution layer and used as the output data of the residual block.

5. The filtering method based on a neural network according to claim 4, characterized in that: The two first convolutional layers of two adjacent residual blocks are respectively the first set convolutional layer and the second set convolutional layer, and / or the two fourth convolutional layers of two adjacent residual blocks are respectively the first set convolutional layer and the second set convolutional layer; or The first convolution layer of the same residual block is the first set convolution layer, and the fourth convolution layer is the second set convolution layer.

6. The neural network-based filtering method according to claim 4 or 5, characterized in that: The convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

7. The neural network-based filtering method according to any one of claims 4 to 6, characterized in that: The first convolution layer includes: two sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of n×n by tensor decomposition, and the two sub-convolution layers include: a convolution layer with a convolution kernel size of 1×n, and a convolution layer with a convolution kernel size of n×1; The convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

8. The neural network-based filtering method according to any one of claims 4 to 7, characterized in that: The first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution, and the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and grouped convolution, and a convolutional layer with a convolution kernel size of n×1 and grouped convolution; The convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

9. The neural network-based filtering method according to any one of claims 4 to 8, characterized in that: The first convolutional layer includes: three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and depthwise separable convolution, and the three sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution, and a convolutional layer with a convolution kernel size of 1×1; The convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

10. The neural network-based filtering method according to any one of claims 4 to 9, characterized in that: The convolution kernel size of the fourth convolutional layer is k×k; or The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k by tensor decomposition; or The fourth convolutional layer includes a sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of k×k through grouped convolution; or The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through depthwise separable convolution; or The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through tensor decomposition and grouped convolution; or The fourth convolutional layer includes three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through tensor decomposition and depthwise separable convolution; Here, k is an integer greater than or equal to 1.

11. The neural network-based filtering method according to any one of claims 1 to 10, characterized in that: The neural network filter also includes: A shallow feature extraction unit, wherein the shallow feature extraction unit includes at least one convolutional layer, and the shallow feature extraction unit is used to extract shallow feature information of the input parameter and input the shallow feature information into the residual unit.

12. The neural network-based filtering method according to claim 11, characterized in that: The input parameters include multiple types of information, the at least one convolutional layer includes multiple shallow feature extraction convolutional layers, each type of information corresponds to at least one shallow feature extraction convolutional layer, and the shallow feature extraction convolutional layer is used to extract shallow feature information of the corresponding information.

13. The neural network-based filtering method according to claim 12, characterized in that: The input parameters include multiple information, and the shallow feature extraction unit also includes a connection layer, which is used to connect the multiple information and input the connected multiple information into the at least one convolution layer, and the at least one convolution layer is used to extract shallow feature information of the connected multiple information and input the shallow feature information into the residual unit.

14. The neural network-based filtering method according to any one of claims 1 to 13, characterized in that: The residual unit is for both the image brightness component and the image chrominance component; The neural network filter also includes: feature mapping units for image brightness components and image chrominance components, respectively, for mapping image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component and the feature mapping unit for the image chrominance component respectively process the output data of the residual unit.

15. The neural network-based filtering method according to any one of claims 1 to 14, characterized in that: The residual unit includes residual units for image brightness components and image chrominance components respectively; The neural network filter also includes: feature mapping units for image brightness components and image chrominance components, respectively, for mapping the image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component processes the output data of the residual unit for the image brightness component, and the feature mapping unit for the image chrominance component processes the output data of the residual unit for the image chrominance component.

16. A video encoding method, characterized in that: Applied to an encoding device, the method comprises: generating input parameters of a neural network filter based on a reconstructed image to be filtered; Based on the input parameters, the reconstructed image is processed by the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; Obtaining a filtered image output by the neural network filter for the reconstructed image; A predicted image corresponding to a next frame of image is generated based on the filtered image, and encoding processing is performed on the next frame of video image based on the predicted image corresponding to the next frame of image.

17. A video decoding method, characterized in that: Applied to a decoding device, the method comprises: generating input parameters of a neural network filter based on a reconstructed image to be filtered; Based on the input parameters, filtering the reconstructed image by the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; Obtaining a filtered image output by the neural network filter for the reconstructed image; A predicted image corresponding to a next frame of image is generated based on the filtered image.

18. A filtering device based on a neural network, characterized in that: include: A generating unit configured to generate input parameters of a neural network filter based on the image to be filtered; A processing unit configured to input the input parameters into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; An acquisition unit is configured to acquire a filtered image output by the neural network filter for the image to be filtered.

19. A video encoding device, characterized in that: Applied to an encoding device, the device comprises: A generating unit configured to generate input parameters of a neural network filter based on a reconstructed image to be filtered; A processing unit, configured to process the reconstructed image through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; An acquisition unit configured to acquire a filtered image output by the neural network filter for the reconstructed image; The encoding unit is configured to generate a predicted image corresponding to a next frame image based on the filtered image, and to encode the next frame video image based on the predicted image corresponding to the next frame image.

20. A video decoding device, characterized in that: Applied to a decoding device, the device comprises: A generating unit configured to generate input parameters of a neural network filter based on a reconstructed image to be filtered; A processing unit, configured to filter the reconstructed image through the neural network filter based on the input parameters, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer alternately arranged, the first set convolutional layer is different from the second set convolutional layer; An acquisition unit configured to acquire a filtered image output by the neural network filter for the reconstructed image; The decoding unit is configured to generate a predicted image corresponding to a next frame of image based on the filtered image.

21. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the method as described in any one of claims 1 to 15 or 16 or 17.

22. A computer-readable storage medium, characterized in that: For storing computer programs; The computer program enables a computer to execute the method according to any one of claims 1 to 15 or 16 or 17 above.

23. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 15 or 16 or 17.

Images