Filtering method, encoding method, decoding method, apparatus, computer-readable medium and electronic device

By designing a neural network filter containing multiple residual blocks and parallel settings in video encoding and decoding, the problem of difficulty in improving the filtering effect in loop filtering processing is solved, and improving the video encoding and decoding while reducing the computational complexity is achieved. efficiency.

WO2024239945A1PCT designated stage expired Publication Date: 2024-11-28TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/091116
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-05-06
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

In the field of video encoding and decoding, how to improve the filtering effect in loop filtering processing to improve the encoding and decoding efficiency is an urgent problem. It is difficult for the existing technology to obtain higher filtering effects while reducing the computational complexity of neural network filters. .

Method used

Using a neural network-based filtering method, a plurality of residual blocks containing sequentially connected in a neural network filter are designed in a neural network filter, and at least one of the residual blocks comprises multiple convolutions with different sizes of the convolution kernel. to obtain feature information in multi-level receptive fields, improve the generalization ability of neural network filters and reduce the computational complexity.

Benefits of technology

On the premise of reducing the computational complexity of neural network filters, the filtering effect is improved, thereby improving the encoding and decoding efficiency of videos, achieving higher encoding performance and lower complexity balance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024091116_28112024_PF_FP_ABST
    Figure CN2024091116_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present application are a filtering method, an encoding method, a decoding method, an apparatus, a computer-readable medium and an electronic device. The filtering method based on a neural network comprises: generating input data of a neural network filter on the basis of an image to be filtered, wherein the input data at least includes the image to be filtered; inputting the input data into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks which are connected in sequence, and a first residual block among the plurality of residual blocks includes a plurality of convolution layers which are arranged in parallel and have different convolution kernel sizes; and acquiring a filtered image which is output by means of the neural network filter for the image to be filtered. By means of the technical solution provided in the embodiments of the present application, the effect of filtering can be improved while reducing the operation complexity of a neural network filter, thereby facilitating an improvement in the encoding and decoding efficiency of a video.
Need to check novelty before this filing date? Find Prior Art

Description

Filtering and encoding / decoding method, device, computer-readable medium, and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 19, 2023, with application number 202310576341.2, and invention name “Filtering and encoding and decoding methods, devices, computer-readable media and electronic devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer and communication technology, and more specifically, to a filtering and encoding / decoding method, device, computer-readable medium, and electronic device. Background Art

[0003] In the field of video coding and decoding, after the predicted image and the reconstructed residual image are superimposed to generate a reconstructed image, the reconstructed image will be distorted. Therefore, in order to obtain a higher-quality image, it is usually necessary to perform loop filtering on the reconstructed image. In the loop filtering process, how to improve the filtering effect to enhance the coding and decoding efficiency is a technical problem that needs to be solved urgently.

[0004] Technical content

[0005] The embodiments of the present application provide a filtering and encoding and decoding method, device, computer-readable medium and electronic device, which can improve the filtering effect while reducing the computational complexity of the neural network filter, thereby helping to improve the encoding and decoding efficiency of the video.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0007] According to one aspect of an embodiment of the present application, a neural network-based filtering method is provided, comprising: obtaining input data of a neural network filter, the input data comprising at least the image to be filtered; inputting the input data into the neural network filter, wherein the neural network filter comprises a first residual unit for extracting image feature information, the first residual unit comprising a plurality of residual blocks connected in sequence, a first residual block of the plurality of residual blocks comprising a plurality of convolution layers arranged in parallel and having different convolution kernel sizes; and obtaining a filtered image output by the neural network filter for the image to be filtered.

[0008] According to one aspect of an embodiment of the present application, a video encoding method is provided, comprising: obtaining input data of a neural network filter, the input data comprising at least the reconstructed image to be filtered; inputting the input data into the neural network filter, wherein the neural network filter comprises a first residual unit for extracting image feature information, the first residual unit comprising a plurality of residual blocks connected in sequence, a first residual block of the plurality of residual blocks comprising a plurality of convolution layers arranged in parallel and having different convolution kernel sizes; obtaining a filtered image output by the neural network filter for the reconstructed image; generating a predicted image corresponding to a next frame image based on the filtered image, and encoding the next frame image based on the predicted image corresponding to the next frame image.

[0009] According to one aspect of an embodiment of the present application, a video decoding method is provided, comprising: obtaining input data of a neural network filter, the input data comprising at least the reconstructed image to be filtered; inputting the input data into the neural network filter, wherein the neural network filter comprises a first residual unit for extracting image feature information, the first residual unit comprising a plurality of residual blocks connected in sequence, a first residual block of the plurality of residual blocks comprising a plurality of convolution layers arranged in parallel and having different convolution kernel sizes; obtaining a filtered image output by the neural network filter for the reconstructed image; generating a predicted image corresponding to a next frame image based on the filtered image, and decoding a video code stream based on the predicted image corresponding to the next frame image.

[0010] According to one aspect of an embodiment of the present application, a neural network-based filtering device is provided, including: a generation unit, configured to obtain input data of a neural network filter, wherein the input data at least includes the image to be filtered; a processing unit, configured to input the input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of residual blocks connected in sequence, and a first residual block of the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and with different convolution kernel sizes; an acquisition unit, configured to obtain a filtered image output by the neural network filter for the image to be filtered.

[0011] According to one aspect of an embodiment of the present application, a video encoding device is provided, including: a generation unit, configured to obtain input data of a neural network filter, the input data at least including the reconstructed image to be filtered; a processing unit, configured to input the input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of residual blocks connected in sequence, and a first residual block of the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and with different convolution kernel sizes; an acquisition unit, configured to obtain a filtered image output by the neural network filter for the reconstructed image; an encoding unit, configured to generate a predicted image corresponding to a next frame image based on the filtered image, and to encode the next frame image based on the predicted image corresponding to the next frame image.

[0012] According to one aspect of an embodiment of the present application, a video decoding device is provided, comprising: a generating unit configured to obtain input data of a neural network filter, wherein the input data at least includes the reconstructed image to be filtered; a processing unit configured to input the input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of residual blocks connected in sequence, and a first residual block of the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and having different convolution kernel sizes; an acquiring unit configured to obtain a filtered image output by the neural network filter for the reconstructed image; and a decoding unit configured to generate a predicted image corresponding to a next frame image based on the filtered image, and to decode a video code stream based on the predicted image corresponding to the next frame image.

[0013] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the above embodiment is implemented.

[0014] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the method described in the above embodiment.

[0015] According to one aspect of an embodiment of the present application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, causing the electronic device to perform the methods provided in the various optional embodiments described above.

[0016] In the technical solutions provided in some embodiments of the present application, since the neural network filter contains a residual unit, the residual unit contains multiple residual blocks connected in sequence, and at least one of the residual blocks contains multiple convolution layers arranged in parallel and with different convolution kernel sizes, the neural network filter can obtain feature information on multiple levels of receptive fields through the residual block, thereby improving the generalization ability of the neural network filter, and can improve the filtering effect while reducing the computational complexity of the neural network filter, which is beneficial to improving the encoding and decoding efficiency of the video.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0018] BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a schematic diagram showing an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied;

[0020] FIG2 is a schematic diagram showing the placement of a video encoding device and a video decoding device in a streaming transmission system;

[0021] FIG3 shows a basic flow chart of a video encoder;

[0022] FIG4 shows a schematic diagram of filtering processing based on CNNLF;

[0023] FIG5 shows a schematic diagram of filtering the luminance component and the chrominance component;

[0024] FIG6 shows a schematic diagram of filtering the luminance component and the chrominance component;

[0025] FIG7 shows a flowchart of a filtering method based on a neural network according to some embodiments of the present application;

[0026] FIG8 shows a schematic structural diagram of a residual block according to some embodiments of the present application;

[0027] FIG9 shows a schematic diagram of the decomposition of CPD;

[0028] FIG10 shows a schematic structural diagram of a residual block according to some embodiments of the present application;

[0029] FIG11 shows a flowchart of a video encoding method according to some embodiments of the present application;

[0030] FIG12 shows a flowchart of a video decoding method according to some embodiments of the present application;

[0031] FIG13 shows a schematic structural diagram of a neural network filter according to some embodiments of the present application;

[0032] FIG14 shows a schematic structural diagram of a residual block according to some embodiments of the present application;

[0033] FIG15 shows a schematic structural diagram of a residual block according to some embodiments of the present application;

[0034] FIG16 shows a schematic structural diagram of a residual block according to some embodiments of the present application;

[0035] FIG17 shows a schematic structural diagram of a residual block according to some embodiments of the present application;

[0036] FIG18 shows a block diagram of a filtering device based on a neural network according to some embodiments of the present application;

[0037] FIG19 shows a block diagram of a video encoding apparatus according to some embodiments of the present application;

[0038] FIG20 shows a block diagram of a video decoding apparatus according to some embodiments of the present application;

[0039] FIG21 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] Example embodiments will now be described in a more complete manner with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to these examples; rather, these embodiments are provided to make this application more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art.

[0041] In addition, the features, structures or characteristics described in the present application may be combined in one or more embodiments in any suitable manner. In the following description, there are many specific details so that the embodiments of the present application can be fully understood. However, it will be appreciated by those skilled in the art that when implementing the technical solution of the present application, it is not necessary to use all the detailed features in the embodiments, one or more specific details may be omitted, or other methods, elements, devices, steps, etc. may be adopted.

[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0043] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0044] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0045] FIG1 shows a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present application can be applied.

[0046] 1 , a system architecture 100 includes a plurality of terminal devices that can communicate with each other via, for example, a network 150. For example, the system architecture 100 may include a first terminal device 110 and a second terminal device 120 interconnected via the network 150. In the embodiment of FIG1 , the first terminal device 110 and the second terminal device 120 perform unidirectional data transmission.

[0047] For example, the first terminal device 110 can encode video data (such as a video picture stream captured by the terminal device 110) for transmission to the second terminal device 120 via the network 150. The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device 120 can receive the encoded video data from the network 150, decode the encoded video data to restore the video data, and display the video picture based on the restored video data.

[0048] In some embodiments of the present application, the system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 for performing bidirectional transmission of encoded video data, such as during a video conference. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 may encode video data (e.g., a video picture stream captured by the terminal device) for transmission to the other of the third terminal device 130 and the fourth terminal device 140 via a network 150. Each of the third terminal device 130 and the fourth terminal device 140 may also receive the encoded video data transmitted by the other of the third terminal device 130 and the fourth terminal device 140, decode the encoded video data to recover the video data, and display the video pictures on an accessible display device based on the recovered video data.

[0049] In the embodiment of FIG1 , the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140 may be servers, personal computers, and smartphones, but the principles disclosed herein are not limited thereto. The embodiments disclosed herein are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks, including, for example, wired and / or wireless communication networks, for transmitting encoded video data between the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140. The communication network 150 may exchange data using circuit-switched and / or packet-switched channels. The network may include a telecommunications network, a local area network, a wide area network, and / or the Internet. For the purposes of this application, unless otherwise explained below, the architecture and topology of network 150 may be immaterial to the operations disclosed herein.

[0050] In some embodiments of the present application, FIG2 illustrates the placement of a video encoding device and a video decoding device in a streaming environment. The subject matter disclosed herein is equally applicable to other video-enabled applications, including, for example, video conferencing, digital television, and storing compressed video on digital media such as CDs, DVDs, and memory sticks.

[0051] The streaming system may include an acquisition subsystem 213, which may include a video source 201, such as a digital camera, that creates an uncompressed video picture stream 202. In one embodiment, the video picture stream 202 includes samples captured by the digital camera. The video picture stream 202 is depicted as a thicker line to emphasize the higher data volume of the video picture stream compared to the encoded video data 204 (or the encoded video stream 204). The video picture stream 202 may be processed by an electronic device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or embody various aspects of the disclosed subject matter, as described in greater detail below. The encoded video data 204 (or the encoded video stream 204) is depicted as a thinner line to emphasize the lower data volume of the encoded video data 204 (or the encoded video stream 204), which may be stored on a streaming server 205 for future use. One or more streaming client subsystems, such as client subsystem 206 and client subsystem 208 in FIG2 , can access streaming server 205 to retrieve copies 207 and 209 of encoded video data 204. Client subsystem 206 can include, for example, a video decoding device 210 in electronic device 230. Video decoding device 210 decodes the incoming copy 207 of the encoded video data and generates an output video picture stream 211 that can be presented on a display 212 (e.g., a screen) or another presentation device. In some streaming systems, the encoded video data 204, video data 207, and video data 209 (e.g., video code streams) can be encoded according to certain video encoding / compression standards.

[0052] It should be noted that the electronic device 220 and the electronic device 230 may include other components not shown in the figure. For example, the electronic device 220 may include a video decoding device, and the electronic device 230 may also include a video encoding device.

[0053] In some embodiments of the present application, taking the international video coding standards HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), and China's national video coding standard AVS as examples, after a video frame image is input, the video frame image will be divided into several non-overlapping processing units according to a block size, and each processing unit will perform similar compression operations. This processing unit is called CTU (Coding Tree Unit), or LCU (Largest Coding Unit). The CTU can continue to be divided more finely to obtain one or more basic coding units CU (Coding Unit). CU is the most basic element in a coding link.

[0054] The following introduces some concepts when encoding CU:

[0055] Predictive Coding: Predictive coding includes intra-frame prediction and inter-frame prediction. The original video signal is predicted by a selected reconstructed video signal to produce a residual video signal. The encoder needs to decide which predictive coding mode to use for the current CU and inform the decoder. Intra-frame prediction means the predicted signal comes from an already coded and reconstructed area within the same image; inter-frame prediction means the predicted signal comes from a previously coded image (called a reference image) that is different from the current image.

[0056] Transform & Quantization: After the residual video signal undergoes transformations such as the Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT), the signal is converted to the transform domain, where the coefficients are known as transform coefficients. The transform coefficients are then subjected to a lossy quantization operation, which loses some information, making the quantized signal more suitable for compression. In some video coding standards, more than one transform scheme may be available, so the encoder must select one for the current CU and inform the decoder. The level of quantization is typically determined by the quantization parameter (QP). A larger QP value means that coefficients with a larger value range will be quantized to the same output, which generally results in greater distortion and a lower bitrate. Conversely, a smaller QP value means that coefficients with a smaller value range will be quantized to the same output, which generally results in less distortion and a higher bitrate.

[0057] Entropy Coding or Statistical Coding: The quantized transform domain signal is statistically compressed and encoded based on the frequency of occurrence of each value, and finally a binary (0 or 1) compressed code stream is output. At the same time, the encoding generates other information, such as the selected coding mode, motion vector data, etc., which also need to be entropy coded to reduce the bit rate. Statistical coding is a lossless coding method that can effectively reduce the bit rate required to express the same signal. Common statistical coding methods include variable length coding (VLC) or context-based binary arithmetic coding (CABAC).

[0058] The context-based binary arithmetic coding (CABAC) process consists of three main steps: binarization, context modeling, and binary arithmetic coding. After binarization of the input syntax elements, the binary data can be encoded using either the normal coding mode or the bypass coding mode. The bypass coding mode eliminates the need to assign a specific probability model to each binary bit. Instead, the input binary bit bin values ​​are directly encoded using a simple bypass encoder, speeding up both encoding and decoding. Generally, different syntax elements are not completely independent, and even the same syntax elements have some memory. Therefore, according to conditional entropy theory, conditional coding using other coded syntax elements can further improve coding performance compared to independent encoding or memoryless coding. This coded symbol information used as a condition is called context. In the normal coding mode, the binary bits of the syntax elements are sequentially fed into the context modeler. The encoder assigns an appropriate probability model to each input binary bit based on the values ​​of previously coded syntax elements or binary bits. This process is known as context modeling. The context model corresponding to the syntax element can be located using ctxIdxInc (context index increment) and ctxIdxStart (context index Start). After the bin value and the assigned probability model are fed into the binary arithmetic encoder for encoding, the context model needs to be updated based on the bin value, which is the adaptive process in encoding.

[0059] Loop Filtering: The changed and quantized signal will be reconstructed through inverse quantization, inverse transformation and prediction compensation operations to obtain a reconstructed image. Compared with the original image, due to the influence of quantization, some information of the reconstructed image is different from the original image, that is, the reconstructed image will produce distortion. Therefore, the reconstructed image can be filtered, such as deblocking filter (DB), SAO (Sample Adaptive Offset) or ALF (Adaptive Loop Filter) and other filters, which can effectively reduce the degree of distortion caused by quantization. Since these filtered reconstructed images will be used as a reference for subsequent encoded images to predict future image signals, the above filtering operation is also called loop filtering, that is, filtering operation within the encoding loop.

[0060] In some embodiments of the present application, FIG3 shows a basic flow chart of a video encoder, in which intra-frame prediction is used as an example for explanation. k [x,y] and predicted image signal Perform difference operation to obtain the residual signal u k [x,y], residual signal u k [x,y] is transformed and quantized to obtain the quantized coefficients. The quantized coefficients are entropy coded to obtain the encoded bit stream, and the reconstructed residual signal u′ is obtained through inverse quantization and inverse transformation. k [x,y], predicted image signal and reconstructed residual signal u′ k [x,y] superposition generates reconstructed image signal Reconstructing image signals On the one hand, it is input to the intra-frame mode decision module and the intra-frame prediction module for intra-frame prediction processing, and on the other hand, it is filtered through loop filtering and outputs the filtered image signal s′ k [x,y], filtered image signal s′ k [x,y] can be used as the reference image for the next frame for motion estimation and motion compensation prediction. Then based on the result of motion compensation prediction s′ r [x+m x ,y+m y ] and intra prediction results Get the predicted image signal of the next frame And continue to repeat the above process until the encoding is completed.

[0061] The above-mentioned loop filtering can be implemented based on NNLF (Neural Network Loop Filter, a loop filter based on a neural network). As shown in Figure 4, after the NNLF is trained, the image to be filtered can be input into the trained NNLF to obtain the filtered image. NNLF usually uses a loss function to constrain the filtered image so that it can be restored to the original image as much as possible. The loss function measures the difference between the predicted value and the true value. The greater the loss value between the predicted value and the true value, the greater the difference, and the goal of training is to reduce the loss value. In some embodiments, during training, NNLF can use L1 norm loss function and / or L2 norm loss function to construct the loss function of the model.

[0062] In order to flexibly process the luminance component and chrominance component of an image separately, a neural network-based loop filter (hereinafter referred to as a neural network filter) can also use a luminance and chroma separation (SLC) method.

[0063] For example, as shown in Figure 5, different network structures are used to construct neural network filters for the luminance component (Y) and the chrominance components (Cb and Cr). During training, different filter models can be trained separately for the luminance component (Y) and the chrominance components (Cb and Cr) to improve the filtering performance for the luminance component and the chrominance components. As shown in Figure 6, in the same network structure model, different modules in the network are used to process the luminance component and the chrominance component separately.

[0064] In video encoding and decoding based on neural networks, the neural network filters proposed in related technologies often build an overall network structure based on basic modules such as 3×3 convolution and residual blocks to train neural network filters. For the network structure of the neural network filter, repeatedly stacking 3×3 convolution and residual blocks can bring considerable coding performance, but it introduces excessively high computational complexity; limiting the number of repetitions of 3×3 convolution and residual blocks, that is, specifying a certain computational complexity, makes it difficult to obtain higher coding and decoding performance. Therefore, the technical solution of the embodiment of the present application proposes a new filtering solution based on a neural network, which can obtain higher coding performance without introducing too much complexity, thereby helping to improve the coding and decoding efficiency of the video.

[0065] It's important to note that neural networks fall under the umbrella of artificial intelligence (AI). AI is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also encompasses the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0066] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0067] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. The neural network filter in the embodiment of the present application is a filter based on machine learning / deep learning.

[0068] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:

[0069] FIG7 shows a flowchart of a neural network-based filtering method according to some embodiments of the present application. The neural network-based filtering method can be performed by a device with computing and processing capabilities, such as a terminal device or a server. Referring to FIG7 , the neural network-based filtering method includes at least steps S710 to S730, which are described in detail as follows:

[0070] In step S710, input data of the neural network filter is obtained, where the input data at least includes an image to be filtered.

[0071] In some embodiments of the present application, the image to be filtered can be an image decoded by the video player, or an image captured. Alternatively, the image to be filtered can be a reconstructed image to be filtered, that is, an image generated by superimposing a reconstructed residual image obtained by inverse quantization and inverse transformation and a predicted image. For example, in the process shown in FIG3 , the reconstructed image is the predicted image signal. and reconstructed residual signal u′ k [x,y] Image signal generated by superposition

[0072] In some embodiments, the input data of the neural network filter may also include at least one of the following information: boundary strength information (i.e., boundary strength), slice-level quantization parameter (i.e., sliceQP), basic quantization parameter (i.e., baseQP), frame type of the image to be filtered, and the predicted image corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

[0073] The frame type of the image to be filtered is used to indicate whether the reconstructed image is an I frame, a P frame, or a B frame. In the process shown in FIG3 , the predicted image corresponding to the reconstructed image is

[0074] In step S720, the input data is input into a neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of residual blocks connected in sequence, and the first residual block among the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and having different convolution kernel sizes.

[0075] In an embodiment of the present application, since the first residual block includes multiple convolution layers arranged in parallel and with different convolution kernel sizes, the neural network filter can obtain feature information on multiple levels of receptive fields through the first residual block, thereby improving the generalization ability of the neural network filter, and can improve the filtering effect while reducing the computational complexity of the neural network filter, which is beneficial to improving the encoding and decoding efficiency of the video.

[0076] It should be noted that in convolutional neural networks, the receptive field is the size of the region in the input feature map that maps a single element of each layer's output feature map. Different convolution window sizes (i.e., kernel sizes) process different numbers of input feature map elements, meaning that convolutional layers with different kernel sizes typically have different receptive fields.

[0077] In some embodiments, as shown in Figure 8, the multiple convolutional layers may include: a first convolutional layer 801, a second convolutional layer 802, and the first residual block further includes a first activation function layer 803 connected to the first convolutional layer 801, a second activation function layer 804 connected to the second convolutional layer 802, and a third convolutional layer 805 and a fourth convolutional layer 806 connected in sequence; wherein, the input end of the first convolutional layer 801 is connected to the input end of the second convolutional layer 802 and serves as the input end of the first residual block, the output end of the first activation function layer 803 is connected to the output end of the second activation function layer 804 and connected to the input end of the third convolutional layer 805; the input data of the first residual block and the output data of the fourth convolutional layer 806 are superimposed and used as the output data of the first residual block.

[0078] It should be noted that the first activation function and the second activation function can be PReLU (Parametric Rectified Linear Unit), ReLU (Rectified Linear Unit), GeLU (Gaussian Error Linear Unit), etc.

[0079] In some embodiments, the convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are positive integers and m≠n. For example, the convolution kernel size of the first convolution layer is 3×3, and the convolution kernel size of the second convolution layer is 1×1. In other embodiments of the present application, the values ​​of m and n can be, for example, 1, 3, 5, 7, etc.

[0080] In some embodiments, the first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n by tensor decomposition; the convolution kernel size of the second convolutional layer is m×m, where m and n are positive integers and m≠n.

[0081] In some embodiments, the tensor decomposition method can use CPD (Canonical Polyadic Decomposition). CPD is a classic tensor decomposition method that can be used to reduce the amount of convolution calculations in neural networks. As shown in Figure 9, based on the principle of CPD, a 3×3 convolution can be approximated by a 1×3 convolution and a 3×1 convolution, thereby effectively reducing the complexity of the convolution operation without significantly reducing network performance.

[0082] In some embodiments, the two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n by tensor decomposition may include: a convolutional layer with a convolution kernel size of 1×n, and a convolutional layer with a convolution kernel size of n×1.

[0083] In some embodiments, the first convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through depthwise separable convolution; the convolution kernel size of the second convolutional layer is m×m, where m and n are positive integers and m≠n. For example, the convolution kernel size of the first convolutional layer is 3×3, and the convolution kernel size of the second convolutional layer is 1×1. In other embodiments of the present application, the values ​​of m and n can be, for example, 1, 3, 5, 7, etc.

[0084] Among them, Depth-wise Separable Convolution (DSC) is an improved algorithm of standard convolution. It first convolves each channel separately with a given convolution kernel size and combines the results. This part is called depth-wise convolution. Then, the depth-wise separable convolution uses a 1×1 convolution kernel for ordinary / standard convolution and outputs a feature map. This part is called pointwise convolution. Depth-wise separable convolution DSC can effectively reduce the complexity of convolution operations.

[0085] In some embodiments, the two sub-convolution layers obtained by decomposing a convolution layer with a convolution kernel size of n×n through depthwise separable convolution include: a convolution layer with a convolution kernel size of n×n and performing grouped convolution, and a convolution layer with a convolution kernel size of 1×1.

[0086] In some embodiments, the first convolutional layer includes: three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and depthwise separable convolution; the convolution kernel size of the second convolutional layer is m×m, where m and n are positive integers and m≠n. For example, the convolution kernel size of the first convolutional layer is 3×3, and the convolution kernel size of the second convolutional layer is 1×1. In other embodiments of the present application, the values ​​of m and n can be, for example, 1, 3, 5, 7, etc.

[0087] In some embodiments, a convolution layer with a convolution kernel size of n×n can be first decomposed by depthwise separable convolution to obtain a convolution layer with a convolution kernel size of n×n and grouped convolution, and a convolution layer with a convolution kernel size of 1×1; and then the convolution layer with a convolution kernel size of n×n and grouped convolution can be decomposed by tensor decomposition into a convolution layer with a convolution kernel size of 1×n and grouped convolution, and a convolution layer with a convolution kernel size of n×1 and grouped convolution.

[0088] In some embodiments, the fourth convolutional layer may be a convolutional layer with a convolution kernel size of k×k, where k is an integer greater than or equal to 1, such as 1, 3, 5, 7, etc. Alternatively, the fourth convolutional layer may be a plurality of sub-convolutional layers obtained by decomposing the convolution kernel through tensor decomposition and / or depthwise separable convolution. For example, the fourth convolutional layer may include two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k by tensor decomposition, and the convolution kernel sizes of the two sub-convolutional layers may be 1×k and k×1, respectively; or the fourth convolutional layer may include two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k by depthwise separable convolution, and the two sub-convolutional layers may be a convolutional layer with a convolution kernel size of k×k and grouped convolution, and a convolutional layer with a convolution kernel size of 1×1; or the fourth convolutional layer may include three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k by tensor decomposition and depthwise separable convolution, and the three sub-convolutional layers may be a convolutional layer with a convolution kernel size of 1×k and grouped convolution, a convolution layer with a convolution kernel size of k×1 and grouped convolution, and a convolution layer with a convolution kernel size of 1×1.

[0089] In some embodiments, the residual unit in the neural network filter can be separated by brightness and chrominance, that is, the first residual unit in the neural network filter can be used to extract image feature information of one of the brightness component and the chrominance component of the image to be filtered, and the neural network filter further includes: a second residual unit for extracting image feature information of the other of the brightness component and the chrominance component of the image to be filtered. In other words, the neural network filter includes residual units for the image brightness component and the image chrominance component respectively. In this case, the residual block structure contained in the residual unit for the image brightness component and the residual block structure contained in the residual unit for the image chrominance component can be the same or different.

[0090] For example, the residual unit for the image luminance component (first residual unit) includes the first residual block in the aforementioned embodiment; the residual unit for the image chrominance component (second residual unit) includes at least one of the following: a second residual block having the same structure as the first residual block in the aforementioned embodiment, and other residual blocks other than the first residual block. In some embodiments, the residual block included in the residual unit for the image luminance component (first residual unit) is the first residual block in the aforementioned embodiment; the residual block included in the residual unit for the image chrominance component (second residual unit) can be a residual block with other structures, such as the residual block shown in Figure 10. The residual block shown in Figure 10 includes a convolution layer with a convolution kernel size of 1×1, a PReLU, a convolution layer with a convolution kernel size of 1×1, and a convolution layer with a convolution kernel size of 3×3. The convolution kernel size shown in Figure 10 can be adjusted according to actual needs, and the activation function can also use ReLU or GeLU.

[0091] In some embodiments, the neural network filter may further include: a shallow feature extraction unit, the shallow feature extraction unit including at least one convolution layer, the shallow feature extraction unit being used to extract shallow feature information of the input data and input the shallow feature information into the residual unit, such as the first residual unit and the second residual unit.

[0092] In some embodiments, the neural network filter may further include: a feature mapping unit for mapping image feature information output by the residual unit. In some embodiments, if the residual unit in the neural network filter includes residual units for image luminance components and image chrominance components, respectively, a first feature mapping unit and a second feature mapping unit for the image luminance component and the image chrominance component may be provided, respectively, and the first feature mapping unit for the image luminance component is connected to the first residual unit for the image luminance component, and the second feature mapping unit for the image chrominance component is connected to the second residual unit for the image chrominance component.

[0093] In step S730, a filtered image output by the neural network filter for the reconstructed image is obtained.

[0094] In some embodiments of the present application, during the training phase, sample input data for training a neural network filter is generated based on a sample image (the sample input data is adjusted according to the usage scenario of the neural network filter), and then the obtained sample input data is input into the neural network filter, and the parameters of the neural network filter are adjusted according to the loss value between the output of the neural network filter and the expected filtering result image corresponding to the sample image, and this process is repeated until the neural network filter meets the convergence conditions.

[0095] It should be noted that the technical solution of the embodiment shown in FIG7 can be applied to the loop filter in the video encoding and decoding process, that is, loop filtering is performed in the video encoding and decoding process using the filtering method shown in FIG7. Alternatively, the technical solution of the embodiment shown in FIG7 can also be applied to post-processing of videos or images, that is, filtering images decoded by the video playback end, images captured by the terminal device, or other images.

[0096] Based on the neural network-based filtering method shown in FIG7 , the present application also proposes a video encoding method in an embodiment. This video encoding method can be executed by a device with computing and processing capabilities, such as a terminal device or a server. The specific process is shown in FIG11 , and includes the following steps S1110 to S1140:

[0097] In step S1110, input data of the neural network filter is obtained, where the input data at least includes a reconstructed image to be filtered.

[0098] The specific implementation details of this step can refer to the aforementioned step S710 and will not be repeated here.

[0099] In step S1120, the input data is input into a neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of residual blocks connected in sequence, and the first residual block among the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and having different convolution kernel sizes.

[0100] The specific implementation details of this step can be referred to the aforementioned step S720 and will not be repeated here.

[0101] In step S1130, a filtered image output by the neural network filter for the reconstructed image is obtained.

[0102] The specific implementation details of this step can refer to the aforementioned step S730 and will not be repeated here.

[0103] In step S1140 , a predicted image corresponding to the next frame image is generated based on the filtered image, and encoding processing is performed on the next frame image based on the predicted image corresponding to the next frame image.

[0104] In some embodiments, after obtaining the filtered image, the process shown in Figure 3 can be referred to, that is, the filtered image of the reconstructed image is used as the reference image of the next frame image for motion estimation and motion compensation prediction, and then the predicted image of the next frame image is obtained based on the result of the motion compensation prediction and the intra-frame prediction result, and the process shown in Figure 3 is repeated until the video image encoding is completed.

[0105] Accordingly, based on the neural network-based filtering method shown in FIG7 , an embodiment of the present application further proposes a video decoding method. This video decoding method can be executed by a device with computing and processing capabilities, such as a terminal device or a server. The specific process is shown in FIG12 , and includes the following steps S1210 to S1240:

[0106] In step S1210, input data of the neural network filter is obtained, where the input data at least includes a reconstructed image to be filtered.

[0107] The specific implementation details of this step can refer to the aforementioned step S710 and will not be repeated here.

[0108] In step S1220, the input data is input into a neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of residual blocks connected in sequence, and the first residual block among the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and having different convolution kernel sizes.

[0109] The specific implementation details of this step can be referred to the aforementioned step S720 and will not be repeated here.

[0110] In step S1230, a filtered image output by the neural network filter for the reconstructed image is obtained.

[0111] The specific implementation details of this step can refer to the aforementioned step S730 and will not be repeated here.

[0112] In step S1240 , a predicted image corresponding to the next frame image is generated based on the filtered image, and the video stream is decoded based on the predicted image corresponding to the next frame image.

[0113] In some embodiments, after obtaining the filtered image of the reconstructed image, the filtered image of the reconstructed image can be used as the reference image of the next frame for motion estimation and motion compensation prediction, and then the predicted image of the next frame is obtained based on the result of the motion compensation prediction and the intra-frame prediction result. The predicted image of the next frame is superimposed again with the reconstructed residual signal obtained by inverse quantization and inverse transformation to generate the reconstructed image of the next frame, and this process is repeated to realize the decoding processing of the video code stream.

[0114] The following is a further description of the implementation details not detailed in the embodiments of the present application with reference to FIG13 to FIG17 :

[0115] In the embodiments of this application, a neural network-based filtering solution is proposed. This solution utilizes methods such as luminance and chrominance separation, multi-level receptive fields, and CPD to design a novel neural network filter structure, thereby achieving a good balance between encoding performance and computational complexity. This solution can be applied to video codecs or video pre- and post-processing products.

[0116] In some embodiments, as shown in FIG13 , a structure of a neural network filter in an embodiment of the present application is shown, which includes an input part, a head part, a middle part, a tail part, and an output part.

[0117] In some embodiments, the input part of the neural network filter may include two parts: image information and side information. Among them, the image information includes the reconstructed image to be filtered, the predicted image corresponding to the reconstructed image to be filtered, and the boundary strength, so as to provide the main image content for the filtering of the encoded and decoded reconstructed image. The side information includes sliceQP, baseQP and IPB. Among them, sliceQP and baseQP are used to indicate the quantization parameter and distortion level of the reconstructed image to be filtered, and IPB is used to indicate the encoding type of the reconstructed image to be filtered (i.e., which one of I frame, P frame and B frame).

[0118] It should be noted that the input part of the neural network filter includes but is not limited to the above information. In addition, the reconstructed image to be filtered is essential in the input part of the neural network filter, and other information can be deleted or added according to actual needs.

[0119] In some embodiments, the neural network filter head is used to extract shallow feature information of the input data and may include two 1×3, two 3×1, and a PReLU activation layer. The input data is concatenated and sent to the neural network filter head module for feature extraction to obtain shallow feature representations.

[0120] In some embodiments, the middle part of the neural network filter is an important backbone structure of the network and may include multiple residual blocks (ResBlocks). In some embodiments, the middle part may adopt the form of luminance and chrominance separation, that is, using different modules Y-Part and UV-Part to separately process the luminance component and chrominance component.

[0121] In some embodiments of the present application, the structure of the residual block may be as shown in FIG14 , where 3×3 convolution and 1×1 convolution are jointly used to obtain multi-level receptive field information.

[0122] In some embodiments of the present application, the structure of the residual block may be as shown in FIG15 , where a 3×3 convolution is decomposed into a 3×1 convolution and a 1×3 convolution according to the CPD principle.

[0123] In some embodiments of the present application, the structure of the residual block can be as shown in Figure 16, using depthwise separable convolution (DSC) to decompose a 3×3 convolution into a 3×3 convolution for grouped convolution and a 1×1 convolution. Where g indicates the use of grouped convolution, that is, the convolution within the specified channel range uses the same parameters.

[0124] In some embodiments of the present application, the structure of the residual block can be as shown in Figure 17, using CPD and DSC at the same time to divide a 3×3 convolution into a 1×3 convolution with grouped convolution, a 3×1 convolution with grouped convolution, and a 1×1 convolution. Where g indicates the use of grouped convolution, that is, the convolution within the specified channel range uses the same parameters.

[0125] It should be noted that each residual block included in the Y-Part and UV-Part in the middle part shown in Figure 13 can adopt any of the structures shown in Figures 14 to 17, or some of the residual blocks can adopt any of the structures shown in Figures 14 to 17. For example, if the Y-Part contains four residual blocks, all four of them can adopt the structure shown in Figure 14; or two of them can adopt the structure shown in Figure 15, and the other two can adopt the structure shown in Figure 16.

[0126] In some embodiments, the number and / or structure of residual blocks included in the Y-Part and UV-Part can be the same or different. For example, different residual blocks can be used in the Y-Part and UV-Part in the middle portion. For example, the Y-Part can use one or more residual blocks shown in Figures 14 to 17, while the UV-Part can use another residual block, such as the residual block shown in Figure 10.

[0127] In some embodiments, referring to FIG13 , to correspond to the luminance and chrominance separation in the middle portion, the tail network of the neural network filter may also employ luminance and chrominance separation to obtain filtered images for luminance and chrominance, respectively. In some embodiments, the tail network may include three convolutional layers and a PReLU activation layer to map feature information with a larger number of channels into image information with a smaller number of channels.

[0128] In some embodiments, the final output of the network, i.e., the filtered image, is obtained by adding the reconstructed image to be filtered input by the neural network filter and the residual obtained by network learning. That is, in the neural network filter structure shown in FIG13 , the neural network is used to learn the residual data. In this way, since the amount of residual data is small, the processing efficiency of the neural network can be accelerated. In other embodiments of the present application, the output of the tail network of the neural network filter can also be directly used as the output of the neural network filter, so that the neural network filter learns the difference between the images before and after filtering.

[0129] It should be noted that the structure of the residual block and the size of the convolution kernel in the embodiment of the present application are only examples. In other embodiments of the present application, they can be adjusted according to the actual application scenario.

[0130] The following describes an embodiment of the device of the present application, which can be used to perform the method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method in the above embodiment of the present application.

[0131] FIG18 shows a block diagram of a filtering device based on a neural network according to some embodiments of the present application. The filtering device based on a neural network can be applied to a device with computing and processing functions, such as a terminal device or a server.

[0132] 18 , a neural network-based filtering device 1800 according to some embodiments of the present application includes: a generating unit 1802 , a processing unit 1804 , and an acquiring unit 1806 .

[0133] Among them, the generation unit 1802 is configured to obtain input data of the neural network filter, and the input data at least includes the image to be filtered; the processing unit 1804 is configured to input the input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, and the first residual unit includes a plurality of residual blocks connected in sequence, and the first residual block among the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and with different convolution kernel sizes; the acquisition unit 1806 is configured to obtain the filtered image output by the neural network filter for the image to be filtered.

[0134] The specific functions and structures of the above-mentioned units can be found in the above-mentioned embodiments and will not be described in detail here.

[0135] FIG19 shows a block diagram of a video encoding apparatus according to some embodiments of the present application. The video encoding apparatus may be applied to a device having a computing and processing function, such as a terminal device or a server.

[0136] 19 , a video encoding apparatus 1900 according to some embodiments of the present application includes: a generating unit 1902 , a processing unit 1904 , an acquiring unit 1906 , and an encoding unit 1908 .

[0137] Among them, the generation unit 1902 is configured to obtain input data of the neural network filter, and the input data at least includes the reconstructed image to be filtered; the processing unit 1904 is configured to input the input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, and the first residual unit includes a plurality of residual blocks connected in sequence, and the first residual block among the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and with different convolution kernel sizes; the acquisition unit 1906 is configured to obtain the filtered image output by the neural network filter for the reconstructed image; the encoding unit 1908 is configured to generate a predicted image corresponding to the next frame image based on the filtered image, and encode the next frame image based on the predicted image corresponding to the next frame image.

[0138] FIG20 shows a block diagram of a video decoding apparatus according to some embodiments of the present application. The video decoding apparatus may be applied to a device having a computing and processing function, such as a terminal device or a server.

[0139] 20 , a video decoding apparatus 2000 according to some embodiments of the present application includes: a generating unit 2002 , a processing unit 2004 , an acquiring unit 2006 , and a decoding unit 2008 .

[0140] Among them, the generation unit 2002 is configured to obtain input data of the neural network filter, and the input data at least includes the reconstructed image to be filtered; the processing unit 2004 is configured to input the input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, and the first residual unit includes a plurality of residual blocks connected in sequence, and the first residual block among the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and with different convolution kernel sizes; the acquisition unit 2006 is configured to obtain the filtered image output by the neural network filter for the reconstructed image; the decoding unit 2008 is configured to generate a predicted image corresponding to the next frame image based on the filtered image, and decode the video code stream based on the predicted image corresponding to the next frame image.

[0141] FIG21 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.

[0142] It should be noted that the computer system 2100 of the electronic device shown in FIG21 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0143] As shown in Figure 21, the computer system 2100 may include a central processing unit (CPU) 2101, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 2102 or the program loaded from the storage part 2108 into the random access memory (RAM) 2103, such as the method described in the above embodiment. In the RAM 2103, various programs and data required for system operation are also stored. The CPU 2101, ROM 2102 and RAM 2103 are connected to each other via a bus 2104. An input / output (I / O) interface 2105 is also connected to the bus 2104.

[0144] The following components can be connected to the I / O interface 2105: an input section 2106 including a keyboard, a mouse, and the like; an output section 2107 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 2108 including a hard disk; and a communication section 2109 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 2109 performs communication processing via a network such as the Internet. A drive 2110 is also connected to the I / O interface 2105 as needed. Removable media 2111, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 2110 as needed, so that computer programs read from the removable media can be installed in the storage section 2108 as needed.

[0145] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program is used to perform the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 2109, and / or installed from a removable medium 2111. When the computer program is executed by the central processing unit (CPU) 2101, the various functions defined in the system of the present application are performed.

[0146] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more computer programs, and when the one or more computer programs are executed by the electronic device, the electronic device implements the method described in the above embodiments.

[0147] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0148] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable an electronic device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0149] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

Claims

1. A filtering method based on a neural network filter, performed by a device having a computing processing function, include: Obtaining input data of a neural network filter, wherein the input data at least includes an image to be filtered; Inputting the input data into the neural network filter, wherein the neural network filter comprises a first residual unit for extracting image feature information, the first residual unit comprises a plurality of residual blocks connected in sequence, and a first residual block among the plurality of residual blocks comprises a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes; Obtain a filtered image output by the neural network filter for the image to be filtered.

2. The method according to claim 1, in, The input data also includes at least one of the following information: Boundary strength information, strip-level quantization parameter, basic quantization parameter, frame type of the image to be filtered, and the predicted image corresponding to the reconstructed image when the image to be filtered is a reconstructed image.

3. The method according to claim 1, in, The multiple convolutional layers include a first convolutional layer and a second convolutional layer; the first residual block further includes: a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third convolutional layer and a fourth convolutional layer connected in sequence; Among them, the input end of the first convolutional layer is connected to the input end of the second convolutional layer and serves as the input end of the first residual block, the output end of the first activation function layer is connected to the output end of the second activation function layer and connected to the input end of the third convolutional layer; the input data of the first residual block and the output data of the fourth convolutional layer are superimposed and serve as the output data of the first residual block.

4. The method according to claim 3, in, The convolution kernel size of the first convolution layer is n×n; the convolution kernel size of the second convolution layer is m×m, where m and n are positive integers and m≠n.

5. The method according to claim 3, in, The first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n by tensor decomposition. The convolution kernel size of the second convolutional layer is m×m, where m and n are positive integers and m≠n.

6. The method according to claim 5, in, The two sub-convolutional layers include: a first sub-convolutional layer with a convolutional kernel size of 1×n, and a second sub-convolutional layer with a convolutional kernel size of n×1.

7. The method according to claim 3, in, The first convolutional layer includes: two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through depthwise separable convolution; The convolution kernel size of the second convolution layer is m×m, where m and n are positive integers and m≠n.

8. The method according to claim 7, in, The two sub-convolutional layers include: a third sub-convolutional layer with a convolution kernel size of n×n and performing grouped convolution, and a fourth sub-convolutional layer with a convolution kernel size of 1×1.

9. The method according to claim 3, in, The first convolutional layer includes: three sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of n×n through tensor decomposition and depthwise separable convolution; The convolution kernel size of the second convolution layer is m×m, where m and n are positive integers and m≠n.

10. The method according to claim 9, in, The three sub-convolutional layers include: a fifth sub-convolutional layer with a convolution kernel size of 1×n and performing grouped convolution, a sixth sub-convolutional layer with a convolution kernel size of n×1 and performing grouped convolution, and a seventh sub-convolutional layer with a convolution kernel size of 1×1.

11. The method according to any one of claims 3 to 10, in, The fourth convolutional layer includes any one of the following: A convolution layer with a kernel size of k×k; The convolution layer with a convolution kernel size of k×k is decomposed into two sub-convolution layers by tensor decomposition; The convolution layer with a kernel size of k×k is decomposed into two sub-convolution layers through depth-wise separable convolution; The convolution layer with a kernel size of k×k is decomposed into three sub-convolution layers through tensor decomposition and depth-wise separable convolution; Wherein, k is a positive integer.

12. The method according to any one of claims 1 to 10, in, The first residual unit is used to extract image feature information of one of the brightness component and the chrominance component of the image to be filtered; the neural network filter further includes a second residual unit, which is used to extract image feature information of the other of the brightness component and the chrominance component of the image to be filtered; The second residual unit includes a plurality of residual blocks connected in sequence, and the structures of the plurality of residual blocks are the same as or different from the structure of the residual blocks included in the first residual unit.

13. The method according to claim 12, in, The first residual unit is used to extract image feature information of the brightness component of the image to be filtered, and the second residual unit is used to extract image feature information of the chrominance component of the image to be filtered; the second residual unit includes at least one of the following: a second residual block having the same structure as the first residual block, and other residual blocks except the second residual block.

14. The method according to claim 12, in, The neural network filter also includes: A shallow feature extraction unit, wherein the shallow feature extraction unit includes at least one convolutional layer, and the shallow feature extraction unit is used to extract shallow feature information of the input data and input the shallow feature information into the first residual unit and the second residual unit.

15. The method according to claim 12, The neural network filter also include: A first feature mapping unit is used to map the output of the first residual unit The image feature information is mapped, and the second feature mapping unit is used to map the image feature information output by the second residual unit; wherein the first feature mapping unit is connected to the first residual unit, and the second feature mapping unit is connected to the second residual unit.

16. A video encoding method, performed by a device having a computing processing function, include: Acquire input data of a neural network filter, wherein the input data at least includes a reconstructed image to be filtered; Inputting the input data into the neural network filter, wherein the neural network filter comprises a first residual unit for extracting image feature information, the first residual unit comprises a plurality of residual blocks connected in sequence, and a first residual block among the plurality of residual blocks comprises a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes; Obtaining a filtered image output by the neural network filter for the reconstructed image; A predicted image corresponding to a next frame of image is generated based on the filtered image, and encoding processing is performed on the next frame of image based on the predicted image corresponding to the next frame of image.

17. A video decoding method, performed by a device having a computing processing function, include: Acquire input data of a neural network filter, wherein the input data at least includes the reconstructed image to be filtered; Inputting the input data into the neural network filter, wherein the neural network filter comprises a first residual unit for extracting image feature information, the first residual unit comprises a plurality of residual blocks connected in sequence, and a first residual block among the plurality of residual blocks comprises a plurality of convolutional layers arranged in parallel and having different convolution kernel sizes; Obtaining a filtered image output by the neural network filter for the reconstructed image; A predicted image corresponding to a next frame of image is generated based on the filtered image, and a video code stream is decoded based on the predicted image corresponding to the next frame of image.

18. A filtering device based on a neural network, include: A generating unit configured to obtain input data of a neural network filter, wherein the input data at least includes the image to be filtered; A processing unit configured to input the input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of residual blocks connected in sequence, and a first residual block among the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and having different convolution kernel sizes; An acquisition unit is configured to acquire a filtered image output by the neural network filter for the image to be filtered.

19. A video encoding device, include: A generating unit is configured to obtain input data of the neural network filter, wherein the input data at least includes the Filtered reconstructed image; A processing unit configured to input the input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of residual blocks connected in sequence, and a first residual block among the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and having different convolution kernel sizes; An acquisition unit configured to acquire a filtered image output by the neural network filter for the reconstructed image; The encoding unit is configured to generate a predicted image corresponding to a next frame image based on the filtered image, and to perform encoding processing on the next frame image based on the predicted image corresponding to the next frame image.

20. A video decoding device, include: A generating unit configured to obtain input data of a neural network filter, wherein the input data at least includes the reconstructed image to be filtered; A processing unit configured to input the input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of residual blocks connected in sequence, and a first residual block among the plurality of residual blocks includes a plurality of convolution layers arranged in parallel and having different convolution kernel sizes; An acquisition unit configured to acquire a filtered image output by the neural network filter for the reconstructed image; The decoding unit is configured to generate a predicted image corresponding to a next frame image based on the filtered image, and decode the video code stream based on the predicted image corresponding to the next frame image.

21. A computer readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.

22. An electronic device, include: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the method as claimed in any one of claims 1 to 17.

23. A computer program product, comprising a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the method as claimed in any one of claims 1 to 17.

Citation Information

Patent Citations

  • Method and apparatus for filtering with mode-aware deep learning

    CN111194555A

  • Inter-frame prediction method and device, encoding and decoding method, codec and electronic equipment

    CN114339219A

  • Neural network loop filtering method and device for video coding

    CN115914654A

  • Acceleration method and system for Android platform video call coding

    CN116320492A