Filtering, encoding and decoding method and device and electronic equipment

Abstract

The embodiment of the invention provides a filtering method, a coding method, a decoding method, a filtering device, a coding device and an electronic device. The filtering method based on the neural network comprises the steps of generating an input parameter of a neural network filter based on a to-be-filtered image; the input parameters are input into a neural network filter, the neural network filter comprises a residual unit used for extracting image feature information, the residual unit comprises a plurality of residual blocks connected in sequence, and each residual block comprises a plurality of convolution layers which are arranged in parallel and have different convolution kernel sizes; the residual unit comprises different first set convolution layers and second set convolution layers which are alternately arranged; and obtaining a filtered image for the to-be-filtered image output by the neural network filter. According to the technical scheme provided by the embodiment of the invention, the filtering effect can be improved while the operation complexity of the neural network filter is reduced, so that the video coding and decoding efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular to a filtering and encoding and decoding method, device and electronic equipment. Background Art

[0002] In the field of video coding and decoding, after the predicted image and the reconstructed residual image are superimposed to generate a reconstructed image, the reconstructed image will be distorted. Therefore, in order to obtain a higher quality image, it is usually necessary to perform loop filtering on the reconstructed image. In the loop filtering process, how to improve the filtering effect to improve the coding and decoding efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a filtering and encoding and decoding method, device and electronic device, which can reduce the impact of asymmetry introduced by decomposition and improve the filtering effect while reducing the computational complexity of the neural network filter, thereby helping to improve the encoding and decoding efficiency of the video.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a filtering method based on a neural network is provided, comprising: generating input parameters of a neural network filter based on an image to be filtered; inputting the input parameters into the neural network filter, wherein the neural network filter comprises a residual unit for extracting image feature information, the residual unit comprises a plurality of residual blocks connected in sequence, the residual block comprises at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit comprises a first set convolutional layer and a second set convolutional layer alternately arranged, the first set convolutional layer is different from the second set convolutional layer; and obtaining a filtered image output by the neural network filter for the image to be filtered.

[0006] According to one aspect of an embodiment of the present application, a video encoding method is provided, comprising: generating input parameters of a neural network filter based on a reconstructed image to be filtered; inputting the input parameters into the neural network filter, wherein the neural network filter comprises a residual unit for extracting image feature information, the residual unit comprises a plurality of residual blocks connected in sequence, the residual block comprises at least a plurality of convolution layers arranged in parallel and having different convolution kernel sizes, the residual unit comprises a first set convolution layer and a second set convolution layer alternately arranged, the first set convolution layer is different from the second set convolution layer; obtaining a filtered image output by the neural network filter for the reconstructed image; generating a predicted image corresponding to a next frame image based on the filtered image, and encoding the next frame of video image based on the predicted image corresponding to the next frame image.

[0007] According to one aspect of an embodiment of the present application, a video decoding method is provided, comprising: generating input parameters of a neural network filter based on a reconstructed image to be filtered; inputting the input parameters into the neural network filter, wherein the neural network filter comprises a residual unit for extracting image feature information, the residual unit comprises a plurality of residual blocks connected in sequence, the residual block comprises at least a plurality of convolution layers arranged in parallel and having different convolution kernel sizes, the residual unit comprises a first set convolution layer and a second set convolution layer arranged alternately, the first set convolution layer is different from the second set convolution layer; obtaining a filtered image output by the neural network filter for the reconstructed image; generating a predicted image corresponding to a next frame image based on the filtered image, and decoding a video code stream based on the predicted image corresponding to the next frame image.

[0008] According to one aspect of an embodiment of the present application, a filtering device based on a neural network is provided, comprising: a generating unit, configured to generate input parameters of a neural network filter based on an image to be filtered; a processing unit, configured to input the input parameters into the neural network filter, wherein the neural network filter comprises a residual unit for extracting image feature information, the residual unit comprises a plurality of residual blocks connected in sequence, the residual block comprises at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit comprises a first set convolutional layer and a second set convolutional layer alternately arranged, the first set convolutional layer is different from the second set convolutional layer; an acquiring unit, configured to acquire a filtered image output by the neural network filter for the image to be filtered.

[0009] In some embodiments of the present application, based on the above solution, the input parameter includes at least one of the following information:

[0010] Boundary strength information, strip-level quantization parameter, basic quantization parameter, frame type of the image to be filtered, and the predicted image corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the multiple convolutional layers include a first convolutional layer and a second convolutional layer, the residual block also includes a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third convolutional layer and a fourth convolutional layer connected in sequence, and the output end of the first activation function layer and the output end of the second activation function layer are connected to the input end of the third convolutional layer; wherein the first convolutional layer and the second convolutional layer receive feature information of the input parameters, the feature information of the input parameters includes the image feature information, the first activation function layer and the second activation function layer are the same activation function layer or are two independent activation function layers; the input parameters of the residual block are superimposed on the output data of the fourth convolutional layer and used as the output data of the residual block.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the two first convolution layers of two adjacent residual blocks are respectively the first set convolution layer and the second set convolution layer, and / or the two fourth convolution layers of two adjacent residual blocks are respectively the first set convolution layer and the second set convolution layer; or the first convolution layer of the same residual block is the first set convolution layer, and the fourth convolution layer is the second set convolution layer.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

[0014] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n by tensor decomposition; the convolution kernel size of the second convolutional layer is m×m, m and n are integers greater than or equal to 1 and m≠n.

[0015] In some embodiments of the present application, based on the aforementioned scheme, the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n, and a convolutional layer with a convolution kernel size of n×1.

[0016] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution; the convolution kernel size of the second convolutional layer is m×m, and m and n are integers greater than or equal to 1 and m≠n.

[0017] In some embodiments of the present application, based on the aforementioned scheme, the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, and a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution.

[0018] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and depth-wise separable convolution; the convolution kernel size of the second convolutional layer is m×m, and m and n are integers greater than or equal to 1 and m≠n.

[0019] In some embodiments of the present application, based on the aforementioned scheme, the three sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution, and a convolutional layer with a convolution kernel size of 1×1.

[0020] In some embodiments of the present application, based on the above solution, the convolution kernel size of the fourth convolution layer is k×k; or

[0021] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k by tensor decomposition; or

[0022] The fourth convolutional layer includes a sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of k×k through grouped convolution; or

[0023] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through depthwise separable convolution; or

[0024] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through tensor decomposition and grouped convolution; or

[0025] The fourth convolutional layer includes three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through tensor decomposition and depthwise separable convolution;

[0026] Here, k is an integer greater than or equal to 1.

[0027] In some embodiments of the present application, based on the aforementioned scheme, the neural network filter also includes: a shallow feature extraction unit, the shallow feature extraction unit includes at least one convolutional layer, the shallow feature extraction unit is used to extract shallow feature information of the input parameters, and input the shallow feature information into the residual unit.

[0028] In some embodiments of the present application, based on the aforementioned scheme, the input parameters include multiple information, the at least one convolutional layer includes multiple shallow feature extraction convolutional layers, each type of information corresponds to at least one shallow feature extraction convolutional layer, and the shallow feature extraction convolutional layer is used to extract shallow feature information of the corresponding information.

[0029] In some embodiments of the present application, based on the aforementioned scheme, the input parameters include multiple information, the shallow feature extraction unit also includes a connection layer, the connection layer is used to connect the multiple information, and the connected multiple information is input into the at least one convolutional layer, the at least one convolutional layer is used to extract shallow feature information of the connected multiple information, and input the shallow feature information into the residual unit.

[0030] In some embodiments of the present application, based on the aforementioned solution, the residual unit is for both the image brightness component and the image chrominance component;

[0031] The neural network filter also includes: feature mapping units for image brightness components and image chrominance components, respectively, for mapping image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component and the feature mapping unit for the image chrominance component respectively process the output data of the residual unit.

[0032] In some embodiments of the present application, based on the aforementioned scheme, the residual unit includes residual units for image brightness components and image chrominance components respectively; the neural network filter also includes: feature mapping units for image brightness components and image chrominance components respectively, for mapping image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component processes the output data of the residual unit for the image brightness component, and the feature mapping unit for the image chrominance component processes the output data of the residual unit for the image chrominance component.

[0033] According to one aspect of an embodiment of the present application, a video encoding device is provided, comprising: a generating unit, configured to generate input parameters of a neural network filter based on a reconstructed image to be filtered; a processing unit, configured to input the input parameters to the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, the first set convolution layer is different from the second set convolution layer; an acquiring unit, configured to acquire a filtered image output by the neural network filter for the reconstructed image; an encoding unit, configured to generate a predicted image corresponding to a next frame image based on the filtered image, and encode the next frame of video image based on the predicted image corresponding to the next frame image.

[0034] According to one aspect of an embodiment of the present application, a video decoding device is provided, comprising: a generating unit, configured to generate input parameters of a neural network filter based on a reconstructed image to be filtered; a processing unit, configured to input the input parameters to the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer arranged alternately, the first set convolution layer is different from the second set convolution layer; an acquiring unit, configured to acquire a filtered image output by the neural network filter for the reconstructed image; a decoding unit, configured to generate a predicted image corresponding to a next frame image based on the filtered image, and decode a video code stream based on the predicted image corresponding to the next frame image.

[0035] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the method described in the above embodiments.

[0036] In the technical solutions provided in some embodiments of the present application, since the neural network filter includes a residual unit, the residual unit includes a plurality of residual blocks connected in sequence, and the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the neural network filter can obtain feature information on multi-level receptive fields through the residual block, thereby improving the generalization ability of the neural network filter. The residual unit also includes alternating different convolution layers, so that the influence of the asymmetry introduced by the decomposition can be reduced while reducing the computational complexity of the neural network filter, thereby improving the filtering effect, and thus helping to improve the encoding and decoding efficiency of the video.

[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present application can be applied;

[0039] Figure 2 A schematic diagram showing the placement of a video encoding device and a video decoding device in a streaming transmission system;

[0040] Figure 3 shows a basic flow chart of a video encoder;

[0041] Figure 4 A schematic diagram of filtering based on NNLF is shown;

[0042] Figure 5 A schematic diagram showing a filtering process for a luminance component and a chrominance component is shown;

[0043] Figure 6 A schematic diagram showing a filtering process for a luminance component and a chrominance component is shown;

[0044] Figure 7 A flowchart of a filtering method based on a neural network according to an embodiment of the present application is shown;

[0045] Figure 8 A schematic diagram of the decomposition method according to CPD is shown;

[0046] Fig. 9 A schematic diagram of an alternating decomposition method according to CPD is shown;

[0047] Fig.10 A schematic diagram of the structure of a residual block according to an embodiment of the present application is shown;

[0048] Fig.11 A schematic diagram of the structure of a residual block according to an embodiment of the present application is shown;

[0049] Fig.12 A flowchart of a video encoding method according to an embodiment of the present application is shown;

[0050] Fig.13 A flowchart of a video decoding method according to an embodiment of the present application is shown;

[0051] FIG14 shows a schematic diagram of the structure of a neural network filter according to an embodiment of the present application;

[0052] Fig.15 A schematic diagram of the structure of a neural network filter according to an embodiment of the present application is shown;

[0053] Fig.16 A schematic diagram of the structure of a residual block according to an embodiment of the present application is shown;

[0054] Fig.17 A schematic diagram of the structure of a residual block according to an embodiment of the present application is shown;

[0055] Fig.18 A schematic diagram of the structure of a residual block according to an embodiment of the present application is shown;

[0056] FIG19 shows a schematic diagram of the structure of a neural network filter according to an embodiment of the present application;

[0057] Fig. 20 A schematic diagram of the structure of a neural network filter according to an embodiment of the present application is shown;

[0058] Fig.21 A schematic diagram of the structure of a neural network filter according to an embodiment of the present application is shown;

[0059] Fig. 22 A schematic diagram of the structure of a neural network filter according to an embodiment of the present application is shown;

[0060] Fig.23 A schematic diagram of the structure of a neural network filter according to an embodiment of the present application is shown;

[0061] Fig.24 A schematic diagram of the structure of a neural network filter according to an embodiment of the present application is shown;

[0062] Fig.25 A schematic diagram of the structure of a neural network filter according to an embodiment of the present application is shown;

[0063] Fig.26 A schematic diagram of the structure of a neural network filter according to an embodiment of the present application is shown;

[0064] Fig. 27 A block diagram of a filtering device based on a neural network according to an embodiment of the present application is shown;

[0065] Fig.28 A block diagram of a video encoding device according to an embodiment of the present application is shown;

[0066] Fig.29 A block diagram of a video decoding device according to an embodiment of the present application is shown;

[0067] Fig.30 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0068] The exemplary embodiments are now described in a more comprehensive manner with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be understood as being limited to these examples; on the contrary, the purpose of providing these embodiments is to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0069] In addition, the features, structures or characteristics described in the present application may be combined in one or more embodiments in any suitable manner. In the following description, there are many specific details so that the embodiments of the present application can be fully understood. However, those skilled in the art will appreciate that when implementing the technical scheme of the present application, all the detailed features in the embodiments may not be needed, one or more specific details may be omitted, or other methods, elements, devices, steps, etc. may be adopted.

[0070] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0071] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0072] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0073] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0074] like Figure 1 As shown, the system architecture 100 includes a plurality of terminal devices, and the terminal devices can communicate with each other through, for example, a network 150. For example, the system architecture 100 may include a first terminal device 110 and a second terminal device 120 interconnected through the network 150. Figure 1 In the embodiment of the present invention, the first terminal device 110 and the second terminal device 120 perform unidirectional data transmission.

[0075] For example, the first terminal device 110 may encode video data (e.g., a video picture stream captured by the terminal device 110) for transmission to the second terminal device 120 via the network 150. The encoded video data is transmitted in the form of one or more encoded video code streams. The second terminal device 120 may receive the encoded video data from the network 150, decode the encoded video data to restore the video data, and display the video picture based on the restored video data.

[0076] In one embodiment of the present application, the system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 that perform bidirectional transmission of encoded video data, which may occur during a video conference, for example. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 may encode video data (e.g., a video picture stream collected by the terminal device) to transmit to the other terminal device of the third terminal device 130 and the fourth terminal device 140 through the network 150. Each of the third terminal device 130 and the fourth terminal device 140 may also receive the encoded video data transmitted by the other terminal device of the third terminal device 130 and the fourth terminal device 140, and may decode the encoded video data to restore the video data, and may display the video picture on an accessible display device according to the restored video data.

[0077] exist Figure 1In the embodiment of the present invention, the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140 may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network 150 represents any number of networks that transmit encoded video data between the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140, including, for example, wired and / or wireless communication networks. The communication network 150 may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purpose of the present application, unless explained below, the architecture and topology of the network 150 may be irrelevant to the operation disclosed in the present application.

[0078] In one embodiment of the present application, Figure 2 The placement of the video encoding device and the video decoding device in the streaming environment is shown. The subject matter disclosed in the present application can be equally applicable to other video-supported applications, including, for example, video conferencing, digital TV (television), storing compressed video on digital media including CD, DVD, memory stick, etc.

[0079] The streaming system may include an acquisition subsystem 213, which may include a video source 201 such as a digital camera, which creates an uncompressed video picture stream 202. In an embodiment, the video picture stream 202 includes samples taken by a digital camera. The video picture stream 202 is depicted as a thick line to emphasize the high data volume of the video picture stream compared to the encoded video data 204 (or the encoded video code stream 204), and the video picture stream 202 can be processed by an electronic device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream 202, the encoded video data 204 (or the encoded video code stream 204) is depicted as a thin line to emphasize the lower data volume of the encoded video data 204 (or the encoded video code stream 204), which can be stored on the streaming server 205 for future use. One or more streaming client subsystems, such as Figure 2, can access the streaming server 205 to retrieve the copies 207 and 209 of the encoded video data 204. The client subsystem 206 can include, for example, a video decoding device 210 in the electronic device 230. The video decoding device 210 decodes the incoming copy 207 of the encoded video data and generates an output video picture stream 211 that can be presented on a display 212 (e.g., a display screen) or another presentation device. In some streaming systems, the encoded video data 204, the video data 207, and the video data 209 (e.g., video bitstreams) can be encoded according to certain video encoding / compression standards.

[0080] It should be noted that the electronic device 220 and the electronic device 230 may include other components not shown in the figure. For example, the electronic device 220 may include a video decoding device, and the electronic device 230 may also include a video encoding device.

[0081] In one embodiment of the present application, taking the international video coding standards HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), and China's national video coding standard AVS as examples, after a video frame image is input, the video frame image will be divided into several non-overlapping processing units according to a block size, and each processing unit will perform similar compression operations. This processing unit is called CTU (Coding Tree Unit), or LCU (Largest Coding Unit). CTU can continue to be divided more finely to obtain one or more basic coding units CU (Coding Unit), which is the most basic element in a coding link.

[0082] The following introduces some concepts when encoding CU:

[0083] Predictive Coding: Predictive coding includes intra-frame prediction and inter-frame prediction. The original video signal is predicted by the selected reconstructed video signal to obtain the residual video signal. The encoder needs to decide which predictive coding mode to choose for the current CU and inform the decoder. Intra-frame prediction means that the predicted signal comes from the encoded and reconstructed area in the same image; inter-frame prediction means that the predicted signal comes from other images that have been encoded and different from the current image (called reference images).

[0084] Transform & Quantization: After the residual video signal is transformed by DFT (Discrete Fourier Transform), DCT (Discrete Cosine Transform), etc., the signal is converted into the transform domain, which is called the transform coefficient. The transform coefficient is further subjected to lossy quantization operation, which loses certain information, making the quantized signal conducive to compression expression. In some video coding standards, there may be more than one transform method to choose from, so the encoder also needs to select one of the transform methods for the current CU and inform the decoder. The degree of quantization is usually determined by the quantization parameter (Quantization Parameter, referred to as QP). A larger QP value means that coefficients with a larger value range will be quantized to the same output, which usually results in greater distortion and lower bit rate; on the contrary, a smaller QP value means that coefficients with a smaller value range will be quantized to the same output, which usually results in less distortion and a higher bit rate.

[0085] Entropy Coding or Statistical Coding: The quantized transform domain signal will be statistically compressed and encoded according to the frequency of occurrence of each value, and finally output as a binary (0 or 1) compressed bit stream. At the same time, the encoding generates other information, such as the selected coding mode, motion vector data, etc., which also need to be entropy encoded to reduce the bit rate. Statistical coding is a lossless coding method that can effectively reduce the bit rate required to express the same signal. Common statistical coding methods include variable length coding (VLC) or context-based binary arithmetic coding (CABAC).

[0086] The CABAC process mainly includes three steps: binarization, context modeling, and binary arithmetic coding. After the input syntax elements are binarized, the binary data can be encoded through the conventional coding mode and the bypass coding mode. The bypass coding mode does not need to assign a specific probability model to each binary bit. The input binary bit bin value is directly encoded with a simple bypass encoder to speed up the entire encoding and decoding. In general, different syntax elements are not completely independent, and the same syntax element itself has a certain memory. Therefore, according to the conditional entropy theory, conditional encoding using other encoded syntax elements can further improve the encoding performance compared to independent encoding or memoryless encoding. These encoded symbol information used as conditions are called context. In the conventional coding mode, the binary bits of the syntax elements enter the context modeler sequentially. The encoder assigns a suitable probability model to each input binary bit according to the value of the previously encoded syntax element or binary bit. This process is called context modeling. The context model corresponding to the syntax element can be located through ctxIdxInc (contextindex increment) and ctxIdxStart (context index Start). After the bin value and the assigned probability model are sent to the binary arithmetic encoder for encoding, the context model needs to be updated according to the bin value, which is the adaptive process in encoding.

[0087] Loop Filtering: The changed and quantized signal will be reconstructed through inverse quantization, inverse transformation and prediction compensation operations to obtain a reconstructed image. Compared with the original image, due to the influence of quantization, some information of the reconstructed image is different from the original image, that is, the reconstructed image will produce distortion. Therefore, the reconstructed image can be filtered, such as deblocking filter (DB), SAO (Sample Adaptive Offset) or ALF (Adaptive Loop Filter) and other filters, which can effectively reduce the degree of distortion caused by quantization. Since these filtered reconstructed images will be used as a reference for subsequent encoded images to predict future image signals, the above filtering operation is also called loop filtering, that is, filtering operation within the encoding loop.

[0088] In one embodiment of the present application, Figure 3 A basic flow chart of a video encoder is shown, in which intra-frame prediction is used as an example for explanation.k [x,y] and predicted image signal Perform difference operation to obtain the residual signal u k [x,y], residual signal u k [x,y] is transformed and quantized to obtain the quantized coefficients. The quantized coefficients are entropy coded to obtain the encoded bit stream, and the reconstructed residual signal u' is obtained through inverse quantization and inverse transformation. k [x,y], predicted image signal and reconstructed residual signal u' k [x,y] superposition generates reconstructed image signal Reconstructing image signals On the one hand, it is input to the intra-frame mode decision module and the intra-frame prediction module for intra-frame prediction processing, and on the other hand, it is filtered through loop filtering and outputs the filtered image signal s' k [x,y], filtered image signal s' k [x,y] can be used as the reference image for the next frame for motion estimation and motion compensation prediction. Then based on the result of motion compensation prediction s' r [x+m x ,y+m y ] and intra prediction results Get the predicted image signal of the next frame And continue to repeat the above process until the encoding is completed.

[0089] The above loop filtering can be implemented based on NNLF (Neural Network Loop Filter). Figure 4 As shown, after the NNLF is trained, the image to be filtered can be input into the trained NNLF to obtain the filtered image. NNLF usually uses a loss function to constrain the filtered image so that it can be restored to the original image as much as possible. The loss function measures the difference between the predicted value and the true value. The larger the loss value between the predicted value and the true value, the greater the difference, and the goal of training is to reduce the loss value. Optionally, during training, NNLF can use L1 norm loss function and / or L2 norm loss function to construct the loss function of the model.

[0090] In order to flexibly process the brightness component and the chrominance component of an image separately, a neural network-based loop filter (hereinafter referred to as a neural network filter) may also use a luminance and chrominance separation (SLC) method.

[0091] For example, Figure 5As shown, different network structures are used to construct neural network filters for the luminance component (Y) and the chrominance components (Cb and Cr), and during training, different filter models can be trained separately for the luminance component (Y) and the chrominance components (Cb and Cr) to improve the filtering performance for the luminance component and the chrominance component. Figure 6 As shown, in the same network structure model, different modules in the network are used to process the brightness component and the chrominance component separately.

[0092] In video encoding and decoding based on neural networks, the neural network filters proposed in related technologies often build an overall network structure based on basic modules such as 3×3 convolution and residual blocks to train neural network filters. For the network structure of the neural network filter, repeated stacking of 3×3 convolution and residual blocks can bring considerable coding performance, but it introduces excessively high computational complexity; limiting the number of repetitions of 3×3 convolution and residual blocks, that is, specifying a certain computational complexity, makes it difficult to obtain higher coding and decoding performance. Therefore, the technical solution of the embodiment of the present application proposes a new filtering solution based on a neural network, which can obtain higher coding performance without introducing too much complexity, thereby helping to improve the coding and decoding efficiency of the video.

[0093] It should be noted that neural networks belong to the category of artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0094] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0095] Machine Learning (ML) is a multi-disciplinary cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. The neural network filter in the embodiment of the present application is a filter based on machine learning / deep learning.

[0096] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0097] Figure 7 A flowchart of a filtering method based on a neural network according to an embodiment of the present application is shown. The filtering method based on a neural network can be executed by a device with a computing and processing function, such as a terminal device or a server. Figure 7 As shown, the filtering method based on the neural network includes at least steps S710 to S730, which are described in detail as follows:

[0098] In step S710, input parameters of the neural network filter are generated based on the image to be filtered.

[0099] In one embodiment of the present application, the image to be filtered may be an image decoded by the video player, or may be an image captured. Alternatively, the image to be filtered may be a reconstructed image to be filtered, that is, an image generated by superimposing a reconstructed residual image obtained by inverse quantization and inverse transformation with a predicted image, such as Figure 3 In the process shown, the reconstructed image is the predicted image signal and reconstructed residual signal u' k [x,y] Image signal generated by superposition The decoded reconstructed image to be filtered can be used for playback, or can be used for playback and provide a reference for prediction of subsequent images.

[0100] In some optional embodiments, the input parameters of the neural network filter may include at least one of the following information: boundary strength information, slice-level quantization parameter (sliceQP), basic quantization parameter (baseQP), frame type of the image to be filtered (IPB), and a predicted image (prediction) corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

[0101] The frame type of the image to be filtered is used to indicate whether the reconstructed image is an I frame, a P frame, or a B frame. Figure 3 In the process shown, the predicted image corresponding to the reconstructed image is

[0102] In some optional embodiments, the image to be filtered may also be used as a kind of information in the input parameters.

[0103] In step S720, the input parameters are input into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer arranged alternately, the first set convolution layer is different from the second set convolution layer, for example, the decomposition order of the first set convolution layer is different from the decomposition order of the second set convolution layer.

[0104] Optionally, the first set convolution layer and the second set convolution layer can be decomposed by tensor decomposition, and the tensor decomposition method can adopt CPD (Canonical Polyadic Decomposition). CPD is a classic tensor decomposition method that can be used in neural networks to reduce the amount of convolution calculation. Figure 8 As shown in the figure, according to the principle of CPD, a 3×3 convolution can be approximated by a 1×3 convolution and a 3×1 convolution, thereby effectively reducing the complexity of the convolution operation without significantly reducing the network performance.

[0105] Optionally, the decomposition order of the first set convolutional layer and the decomposition order of the second set convolutional layer are different, that is, CPD decomposition can be used alternately. Fig. 9 As shown, three 3×3 convolutions can be alternately decomposed according to the CPD principle to obtain a 1×3 convolution, a 3×1 convolution, a 3×1 convolution, a 1×3 convolution, a 1×3 convolution and a 3×1 convolution. For example, the first set convolution layer can be a 1×3 convolution connected to a 3×1 convolution, and the second set convolution layer can be a 3×1 convolution connected to a 1×3 convolution; or, the first set convolution layer can be a 3×1 convolution connected to a 1×3 convolution, and the second set convolution layer can be a 1×3 convolution connected to a 3×1 convolution.

[0106] In an embodiment of the present application, since the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the neural network filter can obtain feature information on multi-level receptive fields through the residual block, thereby improving the generalization ability of the neural network filter. Moreover, since the residual unit includes an alternately arranged first set convolutional layer and a second set convolutional layer that can be decomposed by tensor decomposition, and the decomposition order of the first set convolutional layer and the second set convolutional layer is different, it is possible to reduce the influence of the asymmetry introduced by tensor decomposition while reducing the computational complexity of the neural network filter, thereby improving the filtering effect, and thereby helping to improve the encoding and decoding efficiency of the video.

[0107] It should be noted that in convolutional neural networks, the size of the area in which a single element of the output feature map of each network layer is mapped to the input feature map is called the receptive field. If the convolution window size (i.e., the convolution kernel size) is different, the number of elements in the input feature map processed will also be different, that is, convolution layers with different convolution kernel sizes usually have different receptive field sizes.

[0108] In some optional embodiments, such as Fig.10 and Fig.11 As shown, the multiple convolutional layers include a first convolutional layer and a second convolutional layer, and the residual block also includes a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third convolutional layer and a fourth convolutional layer connected in sequence; wherein, the input end of the first convolutional layer is connected to the input end of the second convolutional layer and serves as the input end of the residual block, the first convolutional layer and the second convolutional layer receive feature information of the input parameters, the feature information of the input parameters includes the image feature information, the first activation function layer and the second activation function layer are the same activation function layer or are two independent activation function layers, and the output end of the first activation function layer and the output end of the second activation function layer are connected to the input end of the third convolutional layer; the input parameter of the residual block is superimposed with the output data of the fourth convolutional layer and serves as the output data of the residual block.

[0109] Alternatively, if Fig.10 As shown, the first activation function layer and the second activation function layer may be two independent activation function layers.

[0110] Alternatively, if Fig.11 As shown, the first activation function layer and the second activation function layer can be the same activation function layer.

[0111] It should be noted that the first activation function and the second activation function can be PReLU (Parametric Rectified Linear Unit), ReLU (Rectified Linear Unit), GeLU (Gaussian Error Linear Unit), etc.

[0112] In some optional embodiments, the two first convolutional layers of two adjacent residual blocks are respectively the first set convolutional layer and the second set convolutional layer, and / or the two fourth convolutional layers of two adjacent residual blocks are respectively the first set convolutional layer and the second set convolutional layer; or the first convolutional layer of the same residual block is the first set convolutional layer, and the fourth convolutional layer is the second set convolutional layer. For example, in two adjacent residual blocks, the first convolutional layer of one residual block can be a 1×3 convolution connected to a 3×1 convolution, and the first convolutional layer of the other residual block can be a 3×1 convolution connected to a 1×3 convolution; or, the first convolutional layer of the same residual block can be a 1×3 convolution connected to a 3×1 convolution, and the fourth convolutional layer can be a 3×1 convolution connected to a 1×3 convolution.

[0113] In some optional embodiments, the convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n. For example, the convolution kernel size of the first convolution layer is 3×3, and the convolution kernel size of the second convolution layer is 1×1. In other embodiments of the present application, the values ​​of m and n can be, for example, 1, 3, 5, 7, etc.

[0114] In some optional embodiments, the first convolution layer includes: two sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of n1×n2 by tensor decomposition; the convolution kernel size of the second convolution layer is m1×m2, and m1, m2, n1 and n2 are integers greater than or equal to 1.

[0115] Optionally, the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n1, and a convolutional layer with a convolution kernel size of n2×1.

[0116] Optionally, n1≠n2, for example, the first convolutional layer includes a 3×1 sub-convolutional layer and a 1×2 sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of 3×2 by tensor decomposition.

[0117] Optionally, n1=n2, for example, the first convolutional layer includes a 3×1 sub-convolutional layer and a 1×3 sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of 3×3 by tensor decomposition.

[0118] In some optional embodiments, the first convolution layer includes: two sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution; the convolution kernel size of the second convolution layer is m×m, m and n are integers greater than or equal to 1 and m≠n. For example, the first convolution layer includes a 3×1 sub-convolution layer and a 1×3 sub-convolution layer obtained by decomposing a convolution layer with a convolution kernel size of 3×3, and the convolution kernel size of the second convolution layer is 1×1. In other embodiments of the present application, the values ​​of m and n can be, for example, 1, 3, 5, 7, etc.

[0119] Optionally, the two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution include: the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and grouped convolution, and a convolutional layer with a convolution kernel size of n×1 and grouped convolution.

[0120] In some optional embodiments, the first convolution layer includes: three sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of n×n through tensor decomposition and depthwise separable convolution; the convolution kernel size of the second convolution layer is m×m, m and n are integers greater than or equal to 1 and m≠n. For example, the first convolution layer includes a 3×1 sub-convolution layer and a 1×3 sub-convolution layer obtained by decomposing a convolution layer with a convolution kernel size of 3×3, and the convolution kernel size of the second convolution layer is 1×1. In other embodiments of the present application, the values ​​of m and n can be, for example, 1, 3, 5, 7, etc.

[0121] Among them, Depth-wise Separable Convolution (DSC) is an improved algorithm of standard convolution. It first uses a given convolution kernel size to convolve each channel separately and combines the results. This part is called depth-wise convolution. Then the depth-wise separable convolution uses a 1×1 convolution kernel for ordinary / standard convolution and outputs a feature map. This part is called pointwise convolution. Depth-wise separable convolution DSC can effectively reduce the complexity of convolution operations.

[0122] Optionally, the convolution layer with a convolution kernel size of n×n can be first decomposed by depthwise separable convolution to obtain a convolution layer with a convolution kernel size of n×n and grouped convolution, and a convolution layer with a convolution kernel size of 1×1; and then the convolution layer with a convolution kernel size of n×n and grouped convolution can be decomposed by tensor decomposition into a convolution layer with a convolution kernel size of 1×n and grouped convolution, and a convolution layer with a convolution kernel size of n×1 and grouped convolution.

[0123] In some optional embodiments, the fourth convolutional layer may be a convolutional layer with a convolution kernel size of k×k, where k is an integer greater than or equal to 1, such as 1, 3, 5, 7, etc. Or the fourth convolutional layer may also be a plurality of sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k by tensor decomposition, and the convolution kernel sizes of the two sub-convolutional layers may be 1×k and k×1, respectively; or the fourth convolutional layer may include a sub-convolutional layer obtained by decomposing the convolutional layer with a convolution kernel size of k×k by grouped convolution; or the fourth convolutional layer may include two sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k by depthwise separable convolution, and the two sub-convolutional layers may be convolutional layers with a convolution kernel size of k×k and grouped convolution, and a convolutional layer with a convolution kernel size of 1×1; or the fourth The convolution layer includes two sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of k×k through tensor decomposition and grouped convolution, and the two sub-convolution layers can be a convolution layer with a convolution kernel size of 1×k and grouped convolution, and a convolution layer with a convolution kernel size of k×1 and grouped convolution; or the fourth convolution layer includes three sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of k×k through tensor decomposition and depthwise separable convolution, and the three sub-convolution layers can be a convolution layer with a convolution kernel size of 1×k and grouped convolution, a convolution layer with a convolution kernel size of k×1 and grouped convolution, and a convolution layer with a convolution kernel size of 1×1.

[0124] In some optional embodiments, the residual unit in the neural network filter may be brightness-chrominance separated, that is, the residual unit in the neural network filter includes residual units for image brightness component and image chrominance component, respectively. In this case, the residual block structure contained in the residual unit for the image brightness component may be the same as or different from the residual block structure contained in the residual unit for the image chrominance component.

[0125] In some optional embodiments, the neural network filter may also include: a shallow feature extraction unit, the shallow feature extraction unit includes at least one convolution layer, the shallow feature extraction unit is used to extract shallow feature information of the input parameters and input the shallow feature information into the residual unit.

[0126] Optionally, at least one convolutional layer of the shallow feature extraction unit includes a convolutional layer with a convolution kernel size of r×r, where r is an integer greater than or equal to 1, such as 1, 3, 5, 7, etc. At least one convolutional layer of the shallow feature extraction unit may also be a plurality of sub-convolutional layers decomposed by tensor decomposition and / or depthwise separable convolution or grouped convolution. For example, at least one convolutional layer of the shallow feature extraction unit includes a convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of r×r through grouped convolution; or, at least one convolutional layer of the shallow feature extraction unit includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of r×r through tensor decomposition or depth-wise separable convolution; or, at least one convolutional layer of the shallow feature extraction unit includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of r×r through tensor decomposition and grouped convolution; or, at least one convolutional layer of the shallow feature extraction unit includes three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of r×r through tensor decomposition and depth-wise separable convolution.

[0127] In some optional embodiments, the input parameters include multiple information, at least one convolution layer includes multiple shallow feature extraction convolution layers, each type of information corresponds to at least one shallow feature extraction convolution layer, and the shallow feature extraction convolution layer is used to extract shallow feature information of the corresponding information.

[0128] In some optional embodiments, the input parameters include multiple information, and the shallow feature extraction unit also includes a connection layer, which is used to connect the multiple information and input the connected multiple information into at least one convolution layer, and the at least one convolution layer is used to extract shallow feature information of the connected multiple information and input the shallow feature information into the residual unit.

[0129] In some optional embodiments, the neural network filter may further include: a feature mapping unit for mapping the image feature information output by the residual unit. Optionally, if the residual unit in the neural network filter is for both the image brightness component and the image chrominance component; then feature mapping units for the image brightness component and the image chrominance component may be respectively set; wherein the feature mapping unit for the image brightness component and the feature mapping unit for the image chrominance component are respectively connected to the residual unit, and the feature mapping unit for the image brightness component and the feature mapping unit for the image chrominance component respectively process the output data of the residual unit.

[0130] Optionally, the feature mapping unit includes a convolution layer with a convolution kernel size of s×s, where s is an integer greater than or equal to 1, such as 1, 3, 5, 7, etc. The convolution layer included in the feature mapping unit may also be a plurality of sub-convolution layers obtained by decomposition through tensor decomposition and / or depthwise separable convolution or grouped convolution. For example, the feature mapping unit includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using tensor decomposition; or, the feature mapping unit includes one sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using grouped convolution; or, the feature mapping unit includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using depthwise separable convolution; or, the feature mapping unit includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using tensor decomposition and grouped convolution; three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of s×s by using tensor decomposition and depthwise separable convolution.

[0131] In some optional embodiments, if the residual unit in the neural network filter includes residual units for image brightness components and image chrominance components respectively, then feature mapping units for image brightness components and image chrominance components can be set respectively, and the feature mapping unit for the image brightness component is connected to the residual unit for the image brightness component, the feature mapping unit for the image brightness component processes the output data of the residual unit for the image brightness component, the feature mapping unit for the image chrominance component is connected to the residual unit for the image chrominance component, and the feature mapping unit for the image chrominance component processes the output data of the residual unit for the image brightness component.

[0132] In step S730, a filtered image output by the neural network filter for the reconstructed image is obtained.

[0133] In one embodiment of the present application, the neural network filter needs to use the same parameters as those used in application as input during training. Specifically, in the training phase, input parameters for training the neural network filter are generated based on the sample image (the input parameters are adjusted according to the use scenario of the neural network filter, that is, the same as the parameters used by the neural network filter when it is used), and then the obtained input parameters are input into the neural network filter, and the parameters of the neural network filter are adjusted according to the loss value between the output of the neural network filter and the expected filtering result image corresponding to the sample image, and this process is repeated until the neural network filter meets the convergence condition.

[0134] It should be noted that Figure 7 The technical solution of the embodiment shown can be applied to the loop filter in the video encoding and decoding process, that is, by Figure 7The filtering method shown is performed in loop filtering in the video codec. Figure 7 The technical solution of the illustrated embodiment can also be applied to post-processing of videos or images, that is, filtering images decoded by a video player, images captured by a terminal device, or other images.

[0135] exist Figure 7 Based on the filtering method based on the neural network shown in FIG. 1 , a video encoding method is also proposed in the embodiment of the present application. The video encoding method can be executed by a device with a computing and processing function, such as a terminal device or a server. The specific process is as follows: Fig.12 As shown, the following steps S1210 to S1240 are included:

[0136] In step S1210, input parameters of the neural network filter are generated based on the reconstructed image to be filtered.

[0137] The specific implementation details of this step can refer to the aforementioned step S710 and will not be repeated here.

[0138] In step S1220, the input parameters are input into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, and the first set convolution layer is different from the second set convolution layer.

[0139] The specific implementation details of this step can refer to the aforementioned step S720 and will not be repeated here.

[0140] In step S1230, a filtered image output by the neural network filter for the reconstructed image is obtained.

[0141] The specific implementation details of this step can refer to the aforementioned step S730 and will not be repeated here.

[0142] In step S1240, a predicted image corresponding to a next frame of image is generated based on the filtered image, and encoding processing is performed on the next frame of video image based on the predicted image corresponding to the next frame of image.

[0143] Optionally, after obtaining the filtered image, you can refer to Figure 3 The process shown in FIG. 1 is to use the filtered image of the reconstructed image as the reference image of the next frame image for motion estimation and motion compensation prediction, and then obtain the predicted image of the next frame image based on the result of the motion compensation prediction and the intra-frame prediction result, and continue to repeat Figure 3 The process shown in is continued until the encoding of the video image is completed.

[0144] Accordingly, in Figure 7 Based on the filtering method based on the neural network shown in FIG. 1 , a video decoding method is also proposed in the embodiment of the present application. The video decoding method can be executed by a device with a computing and processing function, such as a terminal device or a server. The specific process is as follows: Fig.13 As shown, the following steps S1310 to S1340 are included:

[0145] In step S1310, input parameters of the neural network filter are generated based on the reconstructed image to be filtered.

[0146] The specific implementation details of this step can refer to the aforementioned step S710 and will not be repeated here.

[0147] In step S1320, the input parameters are input into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer alternately arranged, and the first set convolutional layer is different from the second set convolutional layer.

[0148] The specific implementation details of this step can refer to the aforementioned step S720 and will not be repeated here.

[0149] In step S1330, a filtered image output by the neural network filter for the reconstructed image is obtained.

[0150] The specific implementation details of this step can refer to the aforementioned step S730 and will not be repeated here.

[0151] In step S1340, a predicted image corresponding to the next frame image is generated based on the filtered image, and the video code stream is decoded based on the predicted image corresponding to the next frame image.

[0152] Optionally, after obtaining the filtered image of the reconstructed image, the filtered image of the reconstructed image can be used as a reference image for the next frame for motion estimation and motion compensation prediction, and then the predicted image of the next frame is obtained based on the result of the motion compensation prediction and the intra-frame prediction result. The predicted image of the next frame is superimposed on the reconstructed residual signal obtained by inverse quantization and inverse transformation to generate the reconstructed image of the next frame, and this process is repeated to realize the decoding processing of the video code stream.

[0153] The following combination Figures 14 to 26 , the implementation details not detailed in the embodiments of the present application are described again:

[0154] In the embodiment of the present application, a filtering scheme based on a neural network is proposed, which can design a novel neural network filter structure by using brightness and chromaticity separation, multi-level receptive field, alternating CPD, etc., so as to achieve a better balance between coding performance and computational complexity. The scheme can be applied to video codecs or video pre- and post-processing products.

[0155] In some optional embodiments, such as Fig.14 and Fig.15 The structure of a neural network filter in an embodiment of the present application is shown, which includes an input part, a head part, a middle part, a tail part and an output part.

[0156] Optionally, the input part of the neural network filter may include two parts: image information and side information. The image information includes the reconstructed image to be filtered, the predicted image corresponding to the reconstructed image to be filtered, and the boundary strength, so as to provide the main image content for the filtering of the encoded and decoded reconstructed image. The side information includes sliceQP, baseQP and IPB. SliceQP and baseQP are used to indicate the quantization parameter and distortion level of the reconstructed image to be filtered, and IPB is used to indicate the encoding type of the reconstructed image to be filtered (i.e., which one of I frame, P frame and B frame).

[0157] It should be noted that the input part of the neural network filter includes but is not limited to the above information. In addition, the reconstructed image to be filtered is indispensable in the input part of the neural network filter, and other information can be deleted or added according to actual needs.

[0158] In some optional embodiments, the head of the neural network filter is used to extract shallow feature information of the input parameters, and may include two 3×3 convolutional layers and a PReLU activation layer. The input parameters are concatenated together and sent to the head module of the neural network filter for feature extraction to obtain shallow feature representation.

[0159] In some optional embodiments, the middle part of the neural network filter is an important backbone structure of the network, and may include multiple residual blocks (ResBlock). Optionally, the middle part may be in the form of brightness and chrominance separation, that is, different modules Y-Part and UV-Part are used to separately process the brightness component and the chrominance component.

[0160] In one embodiment of the present application, the structure of the residual block can be as follows: Fig.16As shown in the figure, according to the CPD principle, a 3×3 convolution is decomposed into a 3×1 convolution and a 1×3 convolution, and the decomposition is performed alternately in the same residual block, so that the first convolution layer in the same residual block is a 1×3 convolution and a 3×1 convolution, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution.

[0161] In one embodiment of the present application, the structure of the residual block can be as follows: Fig.17 As shown in the figure, CPD and GC are used simultaneously to decompose a 3×3 convolution into a 1×3 convolution for grouped convolution and a 3×1 convolution. Among them, g indicates the use of grouped convolution, that is, the convolution within the specified channel range uses the same parameters and is alternately decomposed in the same residual block, so that the first convolution layer in the same residual block is a 1×3 convolution and a 3×1 convolution for grouped convolution, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution.

[0162] In one embodiment of the present application, the structure of the residual block can be as follows: Fig.18 As shown in the figure, using CPD and DSC at the same time, a 3×3 convolution is divided into a 1×3 convolution with grouped convolution, a 3×1 convolution with grouped convolution, and a 1×1 convolution. Among them, g indicates the use of grouped convolution, that is, the convolution within the specified channel range uses the same parameters.

[0163] It should be noted that: Fig.15 Each residual block contained in the Y-Part and UV-Part in the middle part shown in can be adopted Figures 16 to 18 Any structure in , or some of the residual blocks in it adopt Figures 16 to 18 For example, Y-Part contains 4 residual blocks, all of which use Fig.16 The structure shown; or two of them adopt Fig.17 The structure shown, the other two adopt Fig.18 The structure shown.

[0164] Optionally, the number and / or structure of the residual blocks included in the Y-Part and the UV-Part may be the same or different. For example, different residual blocks may be used in the Y-Part and the UV-Part in the middle part, for example, 16 to 16 residual blocks may be used in the Y-Part. Fig.18 One or several residual blocks in , UV-Part uses other residual blocks.

[0165] In some optional embodiments, continue to refer to Fig.15As shown, when the brightness and chromaticity of the middle part are not separated, the same tail structure can be used to obtain the brightness filter image and the chromaticity filter image; when the brightness and chromaticity of the middle part are separated, the tail network of the neural network filter can also take the form of brightness and chromaticity separation to obtain the brightness filter image and the chromaticity filter image respectively. Optionally, the tail can include 2 convolutional layers and a PReLU activation layer to map feature information with a larger number of channels into image information with a smaller number of channels.

[0166] In some optional embodiments, the reconstructed image to be filtered input by the neural network filter and the residual obtained by network learning are added to obtain the final output of the network, that is, the filtered image. Fig.15 In the neural network filter structure shown, the neural network is used to learn the residual data, so that the processing efficiency of the neural network can be accelerated due to the small amount of residual data. In other embodiments of the present application, the output of the tail network of the neural network filter can also be directly used as the output of the neural network filter, so that the neural network filter learns the difference between the image before and after filtering.

[0167] It should be noted that the structure of the residual block and the size of the convolution kernel in the embodiment of the present application are only examples. In other embodiments of the present application, they can be adjusted according to the actual application scenario.

[0168] In some optional embodiments, such as Figures 19 to 26 As shown in the figure, the head of the neural network filter includes multiple 3×3 convolutional layers and multiple PReLU activation layers corresponding to different inputs, and includes a 1×1 convolutional layer and a PReLU activation layer. Multiple PReLU activation layers corresponding to different inputs are connected to the 1×1 convolutional layer.

[0169] In one embodiment of the present application, the structure of the neural network filter is as follows: Fig.19 As shown. The middle part of the neural network filter is composed of multiple first residual blocks and multiple second residual blocks arranged alternately, and the tail of the neural network filter is a single-branch structure. Among them, ResBlockT1 represents the first residual block, ResBlockT2 represents the second residual block, the first convolution layer of ResBlockT1 is to decompose a 3×3 convolution into a 1×3 convolution and a 3×1 convolution according to the CPD principle, and the first convolution layer of ResBlockT2 is to decompose a 3×3 convolution into a 3×1 convolution and a 1×3 convolution according to the CPD principle. The tail contains 2 3×3 convolution layers and a PReLU activation layer.

[0170] In one embodiment of the present application, the structure of the neural network filter is as follows: Fig. 20As shown. The middle part of the neural network filter is composed of a plurality of first residual blocks and second residual blocks that are decomposed according to CPD and GC and arranged alternately, and the tail of the neural network filter is a single-branch structure. Among them, ResBlockT1 represents the first residual block, ResBlockT2 represents the second residual block, the first convolution layer of ResBlockT1 is a 1×3 convolution and a 3×1 convolution obtained by decomposing a 3×3 convolution using CPD and GC, and the first convolution layer of ResBlockT2 is a 3×1 convolution and a 1×3 convolution obtained by decomposing a 3×3 convolution using CPD and GC; the tail contains 2 3×3 convolution layers and a PReLU activation layer.

[0171] In one embodiment of the present application, the structure of the neural network filter is as follows: Fig.21 As shown in the figure, the middle part of the neural network filter is composed of a plurality of residual blocks that are alternately decomposed according to the CPD principle. ResBlockT3 represents the residual block, that is, the first convolution layer of ResBlockT3 is a 1×3 convolution and a 3×1 convolution, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution; and the tail of the neural network filter is a single-branch structure, including 2 3×3 convolution layers and a PReLU activation layer.

[0172] In one embodiment of the present application, the structure of the neural network filter is as follows: Fig. 22 As shown. The middle part of the neural network filter is composed of a plurality of residual blocks obtained by alternately decomposing according to CPD and performing GC. ResBlockT3 represents the residual block, that is, the first convolution layer of ResBlockT3 is a 1×3 convolution and a 3×1 convolution obtained by decomposing a 3×3 convolution using CPD and GC, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution obtained by decomposing a 3×3 convolution using CPD and GC; and the tail of the neural network filter is a single-branch structure, including 2 3×3 convolution layers and a PReLU activation layer.

[0173] In one embodiment of the present application, the structure of the neural network filter is as follows: Fig.23As shown. The middle part of the neural network filter is composed of multiple first residual blocks and multiple second residual blocks arranged alternately, and the tail of the neural network filter is two branches. Among them, ResBlockT1 represents the first residual block, ResBlockT2 represents the second residual block, the first convolution layer of ResBlockT1 is to decompose a 3×3 convolution into a 1×3 convolution and a 3×1 convolution according to the CPD principle, and the first convolution layer of ResBlockT2 is to decompose a 3×3 convolution into a 3×1 convolution and a 1×3 convolution according to the CPD principle; the two branches at the tail include a brightness branch for the brightness component of the image and a chrominance branch for the chrominance component of the image. The brightness branch and the chrominance branch contain 2 3×3 convolution layers and a PReLU activation layer respectively.

[0174] In one embodiment of the present application, the structure of the neural network filter is as follows: Fig.24 As shown. The middle part of the neural network filter is composed of a plurality of first residual blocks and second residual blocks that are decomposed according to CPD and GC and arranged alternately, and the tail of the neural network filter is a double-branch structure. Among them, ResBlockT1 represents the first residual block, ResBlockT2 represents the second residual block, the first convolution layer of ResBlockT1 is a 1×3 convolution and a 3×1 convolution obtained by decomposing a 3×3 convolution using CPD and GC, and the first convolution layer of ResBlockT2 is a 3×1 convolution and a 1×3 convolution obtained by decomposing a 3×3 convolution using CPD and GC; the two branches at the tail include a brightness branch for the brightness component of the image and a chrominance branch for the chrominance component of the image, and the brightness branch and the chrominance branch respectively contain 2 3×3 convolution layers and a PReLU activation layer.

[0175] In one embodiment of the present application, the structure of the neural network filter is as follows: Fig.25 As shown in the figure, the middle part of the neural network filter is composed of a plurality of residual blocks which are alternately decomposed according to the CPD principle. ResBlockT3 represents the residual block, that is, the first convolution layer of ResBlockT3 is a 1×3 convolution and a 3×1 convolution, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution; and the tail of the neural network filter contains two branches, the two branches of the tail include a brightness branch for the brightness component of the image and a chroma branch for the chroma component of the image, and the brightness branch and the chroma branch respectively contain 2 3×3 convolution layers and a PReLU activation layer.

[0176] In one embodiment of the present application, the structure of the neural network filter is as follows: Fig.26As shown. The middle part of the neural network filter is composed of a plurality of third residual blocks obtained by alternate decomposition according to CPD and GC, ResBlockT3 represents the residual block, that is, the first convolution layer of ResBlockT3 is a 1×3 convolution and a 3×1 convolution obtained by grouping convolution by decomposing a 3×3 convolution using CPD and GC, and the fourth convolution layer is a 3×1 convolution and a 1×3 convolution obtained by grouping convolution by decomposing a 3×3 convolution using CPD and GC; and the tail of the neural network filter includes two branches, the two branches of the tail include a brightness branch for the brightness component of the image and a chrominance branch for the chrominance component of the image, and the brightness branch and the chrominance branch respectively include 2 3×3 convolution layers and a PReLU activation layer.

[0177] The following describes an embodiment of the device of the present application, which can be used to execute the method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the above embodiment of the method of the present application.

[0178] Fig.18 A block diagram of a filtering device based on a neural network according to an embodiment of the present application is shown. The filtering device based on a neural network can be applied to a device with a computing and processing function, such as a terminal device or a server.

[0179] Reference Fig. 27 As shown, a neural network-based filtering device 2700 according to an embodiment of the present application includes: a generating unit 2702, a processing unit 2704 and an acquiring unit 2706.

[0180] Among them, the generating unit 2702 is configured to generate input parameters of the neural network filter based on the image to be filtered; the processing unit 2704 is configured to input the input parameters into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, the first set convolution layer is different from the second set convolution layer; the acquiring unit 2706 is configured to acquire a filtered image output by the neural network filter for the image to be filtered.

[0181] In some embodiments of the present application, based on the above solution, the input parameter includes at least one of the following information:

[0182] Boundary strength information, strip-level quantization parameter, basic quantization parameter, frame type of the image to be filtered, and the predicted image corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

[0183] In some embodiments of the present application, based on the aforementioned scheme, the multiple convolutional layers include a first convolutional layer and a second convolutional layer, the residual block also includes a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third convolutional layer and a fourth convolutional layer connected in sequence, and the output end of the first activation function layer and the output end of the second activation function layer are connected to the input end of the third convolutional layer; wherein the first convolutional layer and the second convolutional layer receive feature information of the input parameters, the feature information of the input parameters includes the image feature information, the first activation function layer and the second activation function layer are the same activation function layer or are two independent activation function layers; the input parameters of the residual block are superimposed with the output data of the fourth convolutional layer and used as the output data of the residual block.

[0184] In some embodiments of the present application, based on the aforementioned scheme, the two first convolution layers of two adjacent residual blocks are respectively the first set convolution layer and the second set convolution layer, and / or the two fourth convolution layers of two adjacent residual blocks are respectively the first set convolution layer and the second set convolution layer; or the first convolution layer of the same residual block is the first set convolution layer, and the fourth convolution layer is the second set convolution layer.

[0185] In some embodiments of the present application, based on the aforementioned scheme, the convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

[0186] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution; the convolution kernel size of the second convolutional layer is m×m, m and n are integers greater than or equal to 1 and m≠n.

[0187] In some embodiments of the present application, based on the aforementioned scheme, the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, and a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution.

[0188] In some embodiments of the present application, based on the aforementioned scheme, the first convolutional layer includes: three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and depth-wise separable convolution; the convolution kernel size of the second convolutional layer is m×m, m and n are integers greater than or equal to 1 and m≠n.

[0189] In some embodiments of the present application, based on the aforementioned scheme, the three sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution, and a convolutional layer with a convolution kernel size of 1×1.

[0190] In some embodiments of the present application, based on the above scheme, the convolution kernel size of the fourth convolution layer is k×k; or

[0191] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k by tensor decomposition; or

[0192] The fourth convolutional layer includes a sub-convolutional layer obtained by decomposing the convolutional layer with a convolution kernel size of k×k through grouped convolution; or

[0193] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k through depth-wise separable convolution; or

[0194] The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k through tensor decomposition and grouped convolution; or

[0195] The fourth convolutional layer includes three sub-convolutional layers obtained by decomposing the convolutional layer with a convolution kernel size of k×k through tensor decomposition and depth-wise separable convolution;

[0196] Here, k is an integer greater than or equal to 1.

[0197] In some embodiments of the present application, based on the aforementioned scheme, the residual unit includes residual units for image brightness components and image chrominance components respectively; wherein the residual block structure contained in the residual unit for the image brightness component is the same as or different from the residual block structure contained in the residual unit for the image chrominance component.

[0198] In some embodiments of the present application, based on the aforementioned scheme, the neural network filter also includes: a shallow feature extraction unit, the shallow feature extraction unit includes at least one convolutional layer, the shallow feature extraction unit is used to extract shallow feature information of the input parameters, and input the shallow feature information into the residual unit.

[0199] In some embodiments of the present application, based on the aforementioned scheme, the input parameters include multiple information, at least one convolutional layer includes multiple shallow feature extraction convolutional layers, each type of information corresponds to at least one shallow feature extraction convolutional layer, and the shallow feature extraction convolutional layer is used to extract shallow feature information of the corresponding information.

[0200] In some embodiments of the present application, based on the aforementioned scheme, the input parameters include multiple information, and the shallow feature extraction unit also includes a connection layer, which is used to connect the multiple information and input the connected multiple information into at least one convolutional layer. The at least one convolutional layer is used to extract shallow feature information of the connected multiple information and input the shallow feature information into the residual unit.

[0201] In some embodiments of the present application, based on the aforementioned solution, the residual unit is for both the image brightness component and the image chrominance component;

[0202] The neural network filter also includes: feature mapping units for image brightness components and image chrominance components respectively, which are used to map the image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component and the feature mapping unit for the image chrominance component respectively process the output data of the residual unit.

[0203] In some embodiments of the present application, based on the aforementioned scheme, the residual unit includes residual units for image brightness components and image chrominance components respectively; the neural network filter also includes: feature mapping units for image brightness components and image chrominance components respectively, for mapping image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component processes the output data of the residual unit for the image brightness component, and the feature mapping unit for the image chrominance component processes the output data of the residual unit for the image chrominance component.

[0204] Fig.28 A block diagram of a video encoding device according to an embodiment of the present application is shown. The video encoding device can be applied to a device with a computing and processing function, such as a terminal device or a server.

[0205] Reference Fig.28 As shown, a video encoding device 2800 according to an embodiment of the present application includes: a generating unit 2802 , a processing unit 2804 , an acquiring unit 2806 and an encoding unit 2808 .

[0206] Among them, the generating unit 2802 is configured to generate input parameters of the neural network filter based on the reconstructed image to be filtered; the processing unit 2804 is configured to input the input parameters to the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, the first set convolution layer is different from the second set convolution layer; the acquiring unit 2806 is configured to acquire a filtered image for the reconstructed image output by the neural network filter; the encoding unit 2808 is configured to generate a predicted image corresponding to the next frame image based on the filtered image, and encode the next frame of video image based on the predicted image corresponding to the next frame image.

[0207] Fig. 20 A block diagram of a video decoding device according to an embodiment of the present application is shown. The video decoding device can be applied to a device with a computing and processing function, such as a terminal device or a server.

[0208] Reference Fig.29 As shown, a video decoding device 2900 according to an embodiment of the present application includes: a generating unit 2902 , a processing unit 2904 , an acquiring unit 2906 and a decoding unit 2908 .

[0209] Among them, the generating unit 2902 is configured to generate input parameters of the neural network filter based on the reconstructed image to be filtered; the processing unit 2904 is configured to input the input parameters to the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolution layers arranged in parallel and with different convolution kernel sizes, the residual unit includes a first set convolution layer and a second set convolution layer alternately arranged, the first set convolution layer is different from the second set convolution layer; the acquiring unit 2906 is configured to acquire a filtered image for the reconstructed image output by the neural network filter; the decoding unit 2908 is configured to generate a predicted image corresponding to the next frame image based on the filtered image, and decode the video code stream based on the predicted image corresponding to the next frame image.

[0210] Fig.30 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0211] It should be noted that Fig.30 The computer system 3000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0212] like Fig.30 As shown, the computer system 3000 may include a central processing unit (CPU) 3001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 3002 or the program loaded from the storage part 3008 to the random access memory (RAM) 3003, such as executing the method in the above embodiment. In RAM 3003, various programs and data required for system operation are also stored. CPU 3001, ROM 3002 and RAM 3003 are connected to each other through bus 3004. Input / output (I / O) interface 3005 is also connected to bus 3004.

[0213] The following components can be connected to the I / O interface 3005: an input section 3006 including a keyboard, a mouse, etc.; an output section 3007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 3008 including a hard disk, etc.; and a communication section 3009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 3009 performs communication processing via a network such as the Internet. A drive 3010 is also connected to the I / O interface 3005 as needed. A removable medium 3011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 3010 as needed, so that a computer program read therefrom is installed into the storage section 3008 as needed.

[0214] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program is used to perform the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 3009, and / or installed from a removable medium 3011. When the computer program is executed by a central processing unit (CPU) 3001, various functions defined in the system of the present application are executed.

[0215] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a computer program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0216] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0217] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0218] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more computer programs, and when the above one or more computer programs are executed by an electronic device, the electronic device implements the method in the above embodiment.

[0219] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0220] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable an electronic device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0221] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0222] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A filtering method based on a neural network, characterized in that: include: generating input parameters of a neural network filter based on the image to be filtered; Inputting the input parameters into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; Obtain a filtered image output by the neural network filter for the image to be filtered.

2. The filtering method based on a neural network according to claim 1, characterized in that: The input parameters include at least one of the following information: Boundary strength information, strip-level quantization parameter, basic quantization parameter, frame type of the image to be filtered, and the predicted image corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

3. The filtering method based on a neural network according to claim 1, characterized in that: The multiple convolutional layers include a first convolutional layer and a second convolutional layer, the residual block also includes a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third convolutional layer and a fourth convolutional layer connected in sequence, and the output end of the first activation function layer and the output end of the second activation function layer are connected to the input end of the third convolutional layer; Among them, the first convolution layer and the second convolution layer receive feature information of the input parameters, the feature information of the input parameters includes the image feature information, the first activation function layer and the second activation function layer are the same activation function layer or two independent activation function layers; the input parameters of the residual block are superimposed with the output data of the fourth convolution layer and used as the output data of the residual block.

4. The filtering method based on a neural network according to claim 3, characterized in that: The two first convolutional layers of two adjacent residual blocks are respectively the first set convolutional layer and the second set convolutional layer, and / or the two fourth convolutional layers of two adjacent residual blocks are respectively the first set convolutional layer and the second set convolutional layer; or The first convolution layer of the same residual block is the first set convolution layer, and the fourth convolution layer is the second set convolution layer.

5. The filtering method based on neural network according to claim 3, characterized in that: The convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

6. The filtering method based on neural network according to claim 3, characterized in that: The first convolution layer includes: two sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of n×n by tensor decomposition, and the two sub-convolution layers include: a convolution layer with a convolution kernel size of 1×n, and a convolution layer with a convolution kernel size of n×1; The convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

7. The filtering method based on a neural network according to claim 3, characterized in that: The first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and grouped convolution, and the two sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and grouped convolution, and a convolutional layer with a convolution kernel size of n×1 and grouped convolution; The convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

8. The filtering method based on a neural network according to claim 3, characterized in that: The first convolutional layer includes: three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and depthwise separable convolution, and the three sub-convolutional layers include: a convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, a convolutional layer with a convolution kernel size of n×1 and performing grouped convolution, and a convolutional layer with a convolution kernel size of 1×1; The convolution kernel size of the second convolution layer is m×m, where m and n are integers greater than or equal to 1 and m≠n.

9. The neural network-based filtering method according to any one of claims 3 to 8, characterized in that: The convolution kernel size of the fourth convolutional layer is k×k; or The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k by tensor decomposition; or The fourth convolutional layer includes a sub-convolutional layer obtained by decomposing a convolutional layer with a convolution kernel size of k×k through grouped convolution; or The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through depthwise separable convolution; or The fourth convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through tensor decomposition and grouped convolution; or The fourth convolutional layer includes three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k through tensor decomposition and depth-wise separable convolution; Here, k is an integer greater than or equal to 1.

10. The neural network-based filtering method according to any one of claims 1 to 8, characterized in that: The neural network filter also includes: A shallow feature extraction unit, wherein the shallow feature extraction unit includes at least one convolutional layer, and the shallow feature extraction unit is used to extract shallow feature information of the input parameter and input the shallow feature information into the residual unit.

11. The filtering method based on a neural network according to claim 10, characterized in that: The input parameters include multiple types of information, the at least one convolutional layer includes multiple shallow feature extraction convolutional layers, each type of information corresponds to at least one shallow feature extraction convolutional layer, and the shallow feature extraction convolutional layer is used to extract shallow feature information of the corresponding information.

12. The neural network-based filtering method according to claim 10, characterized in that: The input parameters include multiple information, and the shallow feature extraction unit also includes a connection layer, which is used to connect the multiple information and input the connected multiple information into the at least one convolution layer, and the at least one convolution layer is used to extract shallow feature information of the connected multiple information and input the shallow feature information into the residual unit.

13. The neural network-based filtering method according to any one of claims 1 to 8, characterized in that: The residual unit is for both the image brightness component and the image chrominance component; The neural network filter also includes: feature mapping units for image brightness components and image chrominance components, respectively, for mapping image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component and the feature mapping unit for the image chrominance component respectively process the output data of the residual unit.

14. The neural network-based filtering method according to any one of claims 1 to 8, characterized in that: The residual unit includes residual units for image brightness components and image chrominance components respectively; The neural network filter also includes: feature mapping units for image brightness components and image chrominance components, respectively, for mapping the image feature information output by the residual unit; wherein the feature mapping unit for the image brightness component processes the output data of the residual unit for the image brightness component, and the feature mapping unit for the image chrominance component processes the output data of the residual unit for the image chrominance component.

15. A video encoding method, characterized in that: include: generating input parameters of a neural network filter based on a reconstructed image to be filtered; Inputting the input parameters into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; Obtaining a filtered image output by the neural network filter for the reconstructed image; A predicted image corresponding to a next frame of image is generated based on the filtered image, and encoding processing is performed on the next frame of video image based on the predicted image corresponding to the next frame of image.

16. A video decoding method, characterized in that: include: generating input parameters of a neural network filter based on a reconstructed image to be filtered; Inputting the input parameters into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; Obtaining a filtered image output by the neural network filter for the reconstructed image; A predicted image corresponding to a next frame of image is generated based on the filtered image.

17. A filtering device based on a neural network, characterized in that: include: A generating unit configured to generate input parameters of a neural network filter based on the image to be filtered; A processing unit configured to input the input parameters into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; An acquisition unit is configured to acquire a filtered image output by the neural network filter for the image to be filtered.

18. A video encoding device, characterized in that: include: A generating unit configured to generate input parameters of a neural network filter based on a reconstructed image to be filtered; A processing unit configured to input the input parameters into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer arranged alternately, and the first set convolutional layer is different from the second set convolutional layer; An acquisition unit configured to acquire a filtered image output by the neural network filter for the reconstructed image; The encoding unit is configured to generate a predicted image corresponding to a next frame image based on the filtered image, and to encode the next frame video image based on the predicted image corresponding to the next frame image.

19. A video decoding device, characterized in that: include: A generating unit configured to generate input parameters of a neural network filter based on a reconstructed image to be filtered; A processing unit configured to input the input parameters into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks connected in sequence, the residual block includes at least a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes, the residual unit includes a first set convolutional layer and a second set convolutional layer alternately arranged, the first set convolutional layer is different from the second set convolutional layer; An acquisition unit configured to acquire a filtered image output by the neural network filter for the reconstructed image; The decoding unit is configured to generate a predicted image corresponding to a next frame of image based on the filtered image.

20. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the method as claimed in any one of claims 1 to 16.

Images