Filtering, encoding, and decoding methods and apparatus, electronic devices, and computer programs

Abstract

Embodiments of this application provide a filtering, encoding, and decoding method and apparatus, a computer-readable medium, and an electronic device. The filtering method based on a neural network includes the steps of: generating input data for a neural network filter based on an image awaiting filtering, the input data including at least the image awaiting filtering; inputting the input data into the neural network filter, the neural network filter including a residual unit for extracting image feature information, the residual unit including a plurality of sequentially connected residual blocks, the first residual block of the plurality of residual blocks including a plurality of convolutional layers arranged in parallel and with different convolutional kernel sizes; and obtaining a filtered image of the image awaiting filtering output from the neural network filter. The technical invention in the embodiments of this application can improve the efficiency of video encoding and decoding by improving the filtering effect while reducing the complexity of the neural network filter's computation.

Description

[Technical Field]

[0001] This application claims priority based on a Chinese patent application filed with the China Patent Administration on May 19, 2023, with application number 202310576341.2, and titled "Method, apparatus, computer-readable medium, and electronic device for filtering, encoding, and decoding," the entire contents of which are incorporated herein by reference.

[0002] This application relates to the technical field of computers and communications, and more particularly to methods and apparatus for filtering, encoding and decoding, electronic equipment and computer programs. [Background technology]

[0003] In the field of video encoding and decoding (codecs), distortion can occur in the reconstructed image after it has been generated by superimposing the predicted image and the reconstructed residual image. Therefore, to obtain a relatively good quality image, it is usually necessary to perform loop filtering on the reconstructed image. However, improving the filtering effect of loop filtering and thereby improving the efficiency of encoding and decoding remains a technical problem that needs to be solved. [Overview of the project] [Problems that the invention aims to solve]

[0004] The present invention aims to provide a filtering, encoding, and decoding method and apparatus, electronic device, and computer program that can improve the efficiency of video encoding and decoding by reducing the complexity of the neural network filter's computation while improving the filtering effect.

[0005] Other features and advantages of this application will become apparent from the following detailed description or may be partially acquired through the practice of this application. [Means for solving the problem]

[0006] According to one aspect of the embodiments of this application, a filtering method based on a neural network is provided, which is, The input data for the neural network filter is acquired, and the input data includes at least the image awaiting filtering; The input data is input to the neural network filter, the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block of the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and with different convolutional kernel sizes; and The process includes the step of obtaining filtered images from the images awaiting filtering output from the neural network filter.

[0007] According to one aspect of the embodiments of this application, a video encoding method is provided, which is, The input data for a neural network filter is acquired, and the input data includes at least a reconstructed image awaiting filtering; The input data is input to the neural network filter, the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block of the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and with different convolutional kernel sizes; Obtain a filtered image of the reconstructed image output from the neural network filter; and The steps include generating a predicted image corresponding to the image of the next frame based on the filtered image, and performing encoding processing on the image of the next frame based on the predicted image corresponding to the image of the next frame.

[0008] According to one aspect of the embodiments of this application, a video decoding method is provided, which is, The input data for a neural network filter is acquired, and the input data includes at least a reconstructed image awaiting filtering; The input data is input to the neural network filter, the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block of the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and with different convolutional kernel sizes; Obtain a filtered image of the reconstructed image output from the neural network filter; and The process includes generating a predicted image corresponding to the next frame's image based on the filtered image, and performing a decoding process on the video bitstream based on the predicted image corresponding to the next frame's image.

[0009] According to one aspect of the embodiments of this application, a filtering device based on a neural network is provided, which, A generation unit configured to acquire input data for a neural network filter, wherein the input data includes at least an image awaiting filtering; A processing unit configured to input the aforementioned input data to the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and having different convolutional kernel sizes; and The system includes an acquisition unit configured to acquire filtered images of the images awaiting filtering output from the neural network filter.

[0010] According to one aspect of the embodiments of this application, a video encoding device is provided, which is, A generation unit configured to acquire input data for a neural network filter, wherein the input data includes at least a reconstructed image awaiting filtering; A processing unit configured to input the aforementioned input data to the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and having different convolutional kernel sizes; An acquisition unit configured to acquire a filtered image of the reconstructed image output from the neural network filter; and The system includes an encoding unit that generates a predicted image corresponding to the image of the next frame based on the filtered image, and performs encoding processing on the image of the next frame based on the predicted image corresponding to the image of the next frame.

[0011] According to one aspect of the embodiments of this application, a video decoding device is provided, which is, A generation unit configured to acquire input data for a neural network filter, wherein the input data includes at least a reconstructed image awaiting filtering; A processing unit configured to input the aforementioned input data to the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and having different convolutional kernel sizes; An acquisition unit configured to acquire a filtered image of the reconstructed image output from the neural network filter; and A decoding unit is included that generates a predicted image corresponding to an image of a next frame based on the filtered image, and performs a decoding process on a video bitstream based on the predicted image corresponding to the image of the next frame.

[0012] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, in which a computer program is stored, and when the computer program is executed by a processor, the method described in the above embodiment is realized.

[0013] According to one aspect of an embodiment of the present application, an electronic device is provided, which includes one or more processors and a storage device. The storage device stores one or more computer programs, and when the one or more computer programs are executed by the one or more processors, the method described in the above embodiment is realized on the electronic device.

[0014] According to one aspect of an embodiment of the present application, a computer program product is provided. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. By reading and executing the computer program from the computer-readable storage medium by a processor of an electronic device, the method provided in various selectable embodiments described above is executed on the electronic device.

Advantages of the Invention

[0015] In some embodiments of this application, the neural network filter includes residual units, a plurality of residual blocks in which the residual units are sequentially connected, and a plurality of convolutional layers of different sizes in which at least one of the residual blocks is arranged in parallel. As a result, the neural network filter can acquire feature information in a multi-stage receptive field using the residual blocks, improving the generalizability of the neural network filter, and can also improve the filtering effect while reducing the computational complexity of the neural network filter, thereby improving the efficiency of video encoding and decoding.

[0016] For clarity, the general explanations above and the detailed explanations below are for illustrative and explanatory purposes only and do not limit this application. [Brief explanation of the drawing]

[0017] [Figure 1] This figure shows an exemplary system architecture that may be applied to the technical proposal in the embodiments of this application. [Figure 2] This diagram shows the arrangement of video encoding and video decoding devices in a streaming transmission system. [Figure 3] This is a basic flowchart of a single video encoder. [Figure 4] This figure shows that filtering is performed based on CNNLF. [Figure 5] This figure shows that filtering is performed on the luminance and chromaticity components. [Figure 6] This figure shows that filtering is performed on the luminance and chromaticity components. [Figure 7] This is a flowchart of a neural network-based filtering method in several embodiments of this application. [Figure 8] This figure shows the structure of the residual block in several embodiments of this application. [Figure 9] This diagram shows the decomposition method for CPD. [Figure 10] This figure shows the structure of the residual block in several embodiments of this application. [Figure 11] This is a flowchart of a video encoding method in several embodiments of this application. [Figure 12] This is a flowchart of a video decoding method in several embodiments of this application. [Figure 13] This figure shows the structure of a neural network filter in several embodiments of this application. [Figure 14] This figure shows the structure of the residual block in several embodiments of this application. [Figure 15] This figure shows the structure of the residual block in several embodiments of this application. [Figure 16] This figure shows the structure of the residual block in several embodiments of this application. [Figure 17] This figure shows the structure of the residual block in several embodiments of this application. [Figure 18] This is a block diagram of a neural network-based filtering device in several embodiments of this application. [Figure 19] This is a block diagram of a video encoding device in several embodiments of this application. [Figure 20] This is a block diagram of a video decoding device in several embodiments of this application. [Figure 21] This figure shows the structure of a computer system for realizing an electronic device according to an embodiment of this application. [Modes for carrying out the invention]

[0018] The exemplary embodiments will be described below in a more comprehensive manner with reference to the drawings. However, the exemplary embodiments can be implemented in various ways and should not be understood as being limited to these examples. Rather, the purpose of providing these embodiments is to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0019] Furthermore, the features, structures, or configurations described in this application can be combined in any suitable manner into one or more embodiments. The following description contains many details, which will allow for a thorough understanding of the embodiments of this application. However, as will be understood by those skilled in the art, it is not necessary to use all the details of the embodiments when implementing the technical proposal of this application, and one or more specific details may be omitted, or other methods, elements, apparatus, steps, etc., may be employed.

[0020] The block diagrams shown in the drawings represent only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented using software, or using one or more hardware modules or integrated circuits, or using different networks and / or processors and / or microcontrollers.

[0021] The flowcharts shown in the diagrams are illustrative only and do not necessarily include all content and operations / steps, nor are they necessarily performed in the order described. For example, some operations / steps can be further broken down, and some operations / steps can be merged or partially merged, so the actual order in which they are performed may vary depending on the actual situation.

[0022] In this specification, "plural" refers to two or more. "And / or" is used to describe the relationship between related objects and indicates that there can be three types of relationships. For example, A and / or B can represent three cases: A existing alone, A and B existing simultaneously, and B existing alone. The letter " / " generally indicates that the related objects before and after it have an "or" relationship.

[0023] Figure 1 shows an exemplary system architecture that may be applied to the technical proposal in the embodiment of this application.

[0024] As shown in Figure 1, the system architecture 100 includes a plurality of terminal devices, which can communicate with each other, for example, via a network 150. For example, the system architecture 100 may include a first terminal device 110 and a second terminal device 120 that are connected to each other via the network 150. In the embodiment shown in Figure 1, the first terminal device 110 and the second terminal device 120 perform one-way data transmission.

[0025] For example, the first terminal device 110 can encode video data (for example, a video picture flow collected by the terminal device 110) and transmit it to the second terminal device 120 via the network 150. The encoded video data is transmitted in the form of one or more encoded video bitstreams. The second terminal device 120 receives the encoded video data from the network 150, decodes the encoded video data to recover the video data, and then displays a video picture based on the recovered video data.

[0026] In some embodiments of this application, the system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 that perform bidirectional transmission of encoded video data, the bidirectional transmission of which may occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 can encode video data (e.g., a video picture flow collected by the terminal device) and transmit it via the network 150 to the other terminal device of the third terminal device 130 and the fourth terminal device 140. Each of the third terminal device 130 and the fourth terminal device 140 can further receive the encoded video data transmitted by the other terminal device of the third terminal device 130 and the fourth terminal device 140, decode the encoded video data to recover the video data, and then display the video picture on an accessible display device based on the recovered video data.

[0027] In the embodiment shown in Figure 1, the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140 may be servers, personal computers, and smartphones, but the principles disclosed in this application are not limited thereto. The embodiments disclosed in this application are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. Network 150 is a network that transmits any number of encoded video data between the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140, and includes, for example, wired and / or wireless communication networks. Communication network 150 can exchange data over circuit-switched and / or packet-switched channels. The network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this application, unless otherwise described below, the architecture and topology of network 150 may not be important to the operations disclosed in this application.

[0028] In some embodiments of this application, Figure 2 shows the arrangement of a video encoding device and a video decoding device in a streaming transmission environment. The subject matter disclosed in this application is similarly applicable to other video-supporting applications, such as video conferencing, digital television, and applications that store compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0029] The streaming transmission system may include a collection subsystem 213, which may include a video source 201 such as a digital camera, and the video source generates an uncompressed video picture flow 202. In this embodiment, the video picture flow 202 includes samples captured by the digital camera. Compared to encoded video data 204 (or encoded video bitstream 204), the video picture flow 202 is drawn with thicker lines to highlight the high data volume of the video picture flow, and the video picture flow 202 may be processed by an electronic device 220, which may include a video encoder 203 connected to the video source 201. The video encoder 203 may include hardware, software, or a combination of software and hardware to realize or implement various aspects of the subject matter disclosed in this application, as described in detail below. Compared to the video picture flow 202, the encoded video data 204 (or encoded video bitstream 204) is drawn with a thin line to highlight the low data volume of encoded video data 204 (or encoded video bitstream 204), which may be stored in the streaming transmission server 205 for future use. One or more streaming transmission client subsystems, for example, client subsystems 206 and 208 in Figure 2, can access the streaming transmission server 205 to retrieve copies 207 and 209 of the encoded video data 204. Client subsystem 206 may include, for example, a video decoder 210 in the electronic device 230. The video decoder 210 can decode the input copy 207 of the encoded video data and generate an output video picture flow 211 that can be displayed on an indicator 212 (e.g., a display screen) or another display device. In some streaming transmission systems, encoding can be performed on video data 204, video data 207, and video data 209 (e.g., video bitstream) that have been encoded according to some video encoding / compression standards.

[0030] Note that electronic devices 220 and 230 may include other assemblies not shown. For example, electronic device 220 may include a video decoding device, and electronic device 230 may further include a video encoding device.

[0031] In some embodiments of this application, taking the international video coding standards HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), and the Chinese national video coding standard AVS as examples, after a single video frame image is input, the video frame image is divided into several non-overlapping processing units based on a single block size, and each processing unit undergoes a similar compression operation. This processing unit is called a CTU (Coding Tree Unit) or LCU (Largest Coding Unit). The CTU can be further divided into more detailed parts to obtain one or more basic coding units CU (Coding Unit), and the CU is one of the most fundamental elements in the coding process.

[0032] The following introduces some concepts for encoding CUs.

[0033] Predictive Coding: Predictive coding includes methods such as intra-frame prediction and inter-frame prediction. The original video signal is predicted to a selected reconstructed video signal, and then the residual video signal is obtained. The coding side must decide which predictive coding mode to select for the current CU and notify the decoding side. Intra-frame prediction refers to the predicted signal coming from a coded and reconstructed region within the same image, while inter-frame prediction refers to the predicted signal coming from another coded image (called a reference image) that is different from the current image.

[0034] Transform & Quantization: After a residual video signal undergoes transformation operations such as DFT (Discrete Fourier Transform) and DCT (Discrete Cosine Transform), the signal is transformed into a transformation domain, called the transformation coefficients. The transformation coefficients then undergo a lossy quantization operation, losing a certain amount of information to make the quantized signal more suitable for compressed representation. In some video encoding standards, there may be multiple transformation methods to choose from, so the encoding side must select one of these transformation methods for the current CU and notify the decoding side. The fineness (granularity) of quantization is usually determined by the Quantization Parameter (QP). A larger QP value means that a wider range of coefficients is quantized to the same output, which usually results in greater distortion and a relatively lower bitrate. Conversely, a smaller QP value means that a narrower range of coefficients is quantized to the same output, which usually results in relatively less distortion and can handle a relatively higher bitrate.

[0035] Entropy coding or statistical coding: The quantized transformed domain signal is statistically compressed and coded based on the frequency of each value's occurrence, ultimately resulting in a binarized (0 or 1) compressed bitstream. Other information generated by coding, such as the selected coding mode and motion vector data, also needs to be entropically coded to reduce the bitrate. Statistical coding is a lossless coding method that can effectively reduce the bitrate required to represent the same signal. Common statistical coding methods include Variable Length Coding (VLC) or Content Adaptive Binary Arithmetic Coding (CABAC).

[0036] Context-based binary arithmetic coding (CABAC) processes primarily involve three steps: binarization, context modeling, and binary arithmetic coding. After binarizing the input syntactic elements, the binary data can be coded using either conventional coding mode or bypass coding mode. Bypass coding mode eliminates the need to assign a specific probabilistic model to each binary bit; the input binary bits, i.e., the bin values, are directly coded by a single, simple bypass coder, thereby increasing the overall coding and decoding speed. Generally, different syntactic elements are not entirely independent, and even the same syntactic elements themselves possess a certain degree of memory. Therefore, according to the theory of conditional entropy, conditional coding using other coded syntactic elements can further improve coding performance compared to independent coding or memory-less coding. These coded information used as conditions are called context. In conventional encoding modes, the binary bits of syntactic elements are sequentially fed into the context modeler, and the encoder assigns an appropriate probabilistic model to each input binary bit based on the encoded syntactic elements or the values ​​of the binary bits; this process is context modeling. The ctxIdxInc (context index increment) and ctxIdxStart (context index Start) methods allow positioning to the context model corresponding to the syntactic element. After sending both the bin value and the assigned probabilistic model to the binary arithmetic encoder for encoding, the context model needs to be updated based on the bin value; this is the adaptive process in encoding.

[0037] Loop filtering: A reconstructed image can be obtained by performing inverse quantization, inverse transform, and predictive compensation operations on a transformed and quantized signal. Compared to the original image, the reconstructed image has the effects of quantization, so some information differs from the original image; in other words, distortion can occur in the reconstructed image. Therefore, by performing filtering operations on the reconstructed image, such as DB (Deblocking filter), SAO (Sample Adaptive Offset), and ALF (Adaptive Loop Filter), the distortion generated by quantization can be effectively reduced. These filtered reconstructed images are used as a reference for subsequent encoded images to predict future image signals, so the above filtering operations are also called loop filtering, that is, filtering operations within the encoding loop.

[0038] In some embodiments of this application, Figure 3 shows a basic flowchart of a video encoder, and this flowchart is explained using in-frame prediction as an example. k [x,y] and predicted image signal

[0039]

number

[0040]

number

[0041]

Number

[0042] >> The above loop filtering can be realized based on NNLF (Neural Network Loop Filter, a loop filter based on a neural network). As shown in FIG. 4, after the NNLF is trained, the filtered image can be obtained by inputting the image waiting for filtering into the trained NNLF. The NNLF usually uses a loss function to constrain the filtered image so that it can be restored to the original image as much as possible. The loss function evaluates the difference between the predicted value and the actual value. The larger the loss value between the predicted value and the actual value, the larger the difference. The goal of training is to reduce the loss value. In some embodiments, when training, the NNLF can adopt the L1 norm loss function and / or the L2 norm loss function to construct the loss function of the model.

[0043] To flexibly process the luminance and chromaticity components of an image separately, a neural network-based loop filter (hereinafter abbreviated as neural network filter) can also use the SLC (Separation of Luma and Chroma) method.

[0044] For example, as shown in Figure 5, when constructing and training neural network filters using different network structures for the luminance component (Y) and chromaticity components (Cb and Cr), the filtering performance for the luminance component (Y) and chromaticity component (Cb and Cr) can be improved by training different filter models independently for each component. As shown in Figure 6, the same network structure model processes the luminance component and chromaticity component using different modules within the network, respectively.

[0045] In neural network-based video coding and decoding, the neural network filters proposed in related technologies often construct an overall network structure based on basic modules such as 3x3 convolutions and residual blocks, and then train the neural network filters. Regarding the network structure of the neural network filter, while repeatedly stacking 3x3 convolutions and residual blocks can achieve fairly high coding performance, the computational complexity becomes too high. On the other hand, limiting the number of iterations of 3x3 convolutions and residual blocks, i.e., defining a certain level of computational complexity, makes it difficult to obtain relatively high coding and decoding performance. Therefore, the technical solution in the embodiment of this application proposes a new neural network-based filtering scheme that can improve the efficiency of video coding and decoding by obtaining relatively high coding performance while avoiding the introduction of excessive complexity.

[0046] Furthermore, neural networks belong to the field of Artificial Intelligence (AI), which is a theory, method, technology, and application system that uses digital computers or devices controlled by digital computers to simulate, extend, and augment human intelligence, sense the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that aims to understand the essence of intelligence and create new smart devices that can respond in a similar way to human intelligence. Artificial intelligence studies the design principles and implementation methods of various smart devices, enabling them to have sensing, reasoning, and decision-making functions.

[0047] Artificial intelligence (AI) technology is an interdisciplinary field encompassing a wide range of disciplines, including both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, large-scale data processing technologies, operation / interaction systems, and mechatronics. AI software technologies primarily encompass several areas, such as computer vision, speech processing, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation.

[0048] Machine learning (ML) is a multidisciplinary and interdisciplinary technique encompassing probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specializes in studying how computers can simulate human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structures, and continuously improve performance. Machine learning is at the core of artificial intelligence, a fundamental method for making computers intelligent, and its applications extend to various fields of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and demonstration learning. The neural network filter in the embodiment of this application is a filter based on machine learning / deep learning.

[0049] The following describes in detail the details of how the technical solution described in the embodiment of this application is implemented.

[0050] Figure 7 shows a flowchart of a neural network-based filtering method in several embodiments of this application, and the neural network-based filtering method may be executed by a device with computing processing capabilities, for example, by a terminal device or a server. As shown in Figure 7, the neural network-based filtering method includes at least steps S710 to S730, specifically as follows.

[0051] In step S710, input data for the neural network filter is acquired, and this input data includes at least the image awaiting filtering.

[0052] In some embodiments of this application, the filtering await image may be an image decoded on the video playback side, or a captured image. Alternatively, the filtering await image may be a reconstructed image awaiting filtering, that is, an image generated by superimposing a reconstructed residual image obtained after inverse quantization and inverse transform processing with a predicted image. For example, in the flow shown in Figure 3, the reconstructed image is the predicted image signal.

[0053]

number

[0054] In some embodiments, the input data for the neural network filter may further include at least one of the following pieces of information: boundary strength information, slice quantization parameters (sliceQP), base quantization parameters (baseQP), the frame type of the image awaiting filtering, and the corresponding predicted image when the image awaiting filtering is a reconstructed image.

[0055] Of these, the frame type of the images awaiting filtering is used to indicate whether the reconstructed image is an I-frame, P-frame, or B-frame. In the flow shown in Figure 3, the predicted image corresponding to the reconstructed image is

[0056]

number

[0057] In step S720, input data is input to a neural network filter, which includes a first residual unit for extracting image feature information. This first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among these plurality of residual blocks is arranged in parallel and includes a plurality of convolutional layers with different convolutional kernel sizes.

[0058] In the embodiments of this application, the first residual block is arranged in parallel and includes multiple convolutional layers with different convolutional kernel sizes. This allows the neural network filter to acquire feature information in multi-stage receptive fields using the first residual block, improving the generalizability of the neural network filter. Furthermore, it can improve the filtering effect while reducing the computational complexity of the neural network filter, thereby improving the efficiency of video encoding and decoding.

[0059] In a convolutional neural network, the size of the region where one element of the output feature map of each network layer is mapped to the input feature map is called the receptive field. When the size of the convolution window (i.e., the size of the convolutional kernel) is different, the number of elements of the input feature map processed is also different. In other words, convolutional layers with different convolutional kernel sizes usually have receptive fields of different sizes.

[0060] In some embodiments, as shown in Figure 8, the plurality of convolutional layers may include a first convolutional layer 801 and a second convolutional layer 802, and the first residual block further includes a first activation function layer 803 connected to the first convolutional layer 801, a second activation function layer 804 connected to the second convolutional layer 802, and a third convolutional layer 805 and a fourth convolutional layer 806 connected sequentially, wherein the input terminal of the first convolutional layer 801 is connected to the input terminal of the second convolutional layer 802 and is the input terminal of the first residual block, the output terminal of the first activation function layer 803 is connected to the output terminal of the second activation function layer 804 and is connected to the input terminal of the third convolutional layer 805, and the superposition of the input data of the first residual block and the output data of the fourth convolutional layer 806 is the output data of the first residual block.

[0061] The first and second activation functions may also be PReLU (Parametric Rectified Linear Unit), ReLU (Rectified Linear Unit), GeLU (Gaussian Error Linear Unit), etc.

[0062] In some embodiments, the size of the convolution kernel of the first convolutional layer is n × n, and the size of the convolution kernel of the second convolutional layer is m × m, where m and n are positive integers and m ≠ n. For example, the size of the convolution kernel of the first convolutional layer is 3 × 3, and the size of the convolution kernel of the second convolutional layer is 1 × 1. In other embodiments of this application, the values ​​of m and n may be, for example, 1, 3, 5, 7, etc.

[0063] In some embodiments, the first convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolutional kernel size of n×n using a tensor decomposition method, and the convolutional kernel size of the second convolutional layer is m×m, where m and n are positive integers and m≠n.

[0064] In some embodiments, the tensor decomposition method may be CPD (Canonical Polyadic Decomposition). CPD is a traditional tensor decomposition method that can be used in neural networks to reduce the computational complexity of convolutions. As shown in Figure 9, according to the principle of CPD, one 3x3 convolution can be approximately represented by one 1x3 convolution and one 3x1 convolution, thereby effectively reducing the complexity of convolution operations without degrading network performance.

[0065] In some embodiments, the two subconvolutional layers obtained by decomposing a convolutional layer with a convolutional kernel size of n×n using the tensor decomposition method may include a convolutional layer with a convolutional kernel size of 1×n and a convolutional layer with a convolutional kernel size of n×1.

[0066] In some embodiments, the first convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolutional kernel size of n×n using depth-separable convolution, and the convolutional kernel size of the second convolutional layer is m×m, where m and n are positive integers and m≠n. For example, the convolutional kernel size of the first convolutional layer is 3×3, and the convolutional kernel size of the second convolutional layer is 1×1. In other embodiments of this application, the values ​​of m and n may be, for example, 1, 3, 5, 7, etc.

[0067] Among these, Depth-wise Separable Convolution (DSC) is an improved logicum of standard convolution. It first performs a convolution on each channel using a given size of convolution kernel, and then combines the results. This part is called depth-wise convolution. Next, Depth-wise Separable Convolution performs a normal / standard convolution using a 1x1 convolution kernel and outputs a feature map. This part is called point-wise convolution. Depth-wise Separable Convolution (DSC) can effectively reduce the complexity of the convolution operation.

[0068] In some embodiments, two subconvolutional layers obtained by decomposing a convolutional layer with a convolutional kernel size of n×n using depth-separable convolution include a convolutional layer with a convolutional kernel size of n×n and in which settling convolution is performed, and a convolutional layer with a convolutional kernel size of 1×1.

[0069] In some embodiments, the first convolutional layer includes three sub-convolutional layers obtained by decomposing a convolutional layer with a convolutional kernel size of n×n using tensor decomposition and depth-separable convolution, the convolutional kernel size of the second convolutional layer is m×m, where m and n are positive integers and m≠n. For example, the convolutional kernel size of the first convolutional layer is 3×3, and the convolutional kernel size of the second convolutional layer is 1×1. In other embodiments of this application, the values ​​of m and n may be, for example, 1, 3, 5, 7, etc.

[0070] In some embodiments, a convolutional layer with a convolution kernel size of n×n is first decomposed using depth-separable convolution to obtain a convolutional layer with a convolution kernel size of n×n and in which grouping convolution is performed, and a convolutional layer with a convolution kernel size of 1×1. Subsequently, using tensor decomposition, the convolutional layer with a convolution kernel size of n×n and in which grouping convolution can be decomposed into a convolutional layer with a convolution kernel size of 1×n and in which grouping convolution is performed, and a convolutional layer with a convolution kernel size of n×1 and in which grouping convolution is performed.

[0071] In some embodiments, the above-described fourth convolutional layer may be a convolutional layer with a convolution kernel size of k×k, where k is an integer greater than or equal to 1, such as 1, 3, 5, 7, etc. Alternatively, the above-described fourth convolutional layer may be a plurality of sub-convolutional layers obtained by decomposition using tensor decomposition and / or depth-separable convolution. For example, the fourth convolutional layer may include two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k using the tensor decomposition method, and the convolution kernel sizes of these two sub-convolutional layers may be 1×k and k×1, respectively. Or, the fourth convolutional layer may include two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k using depth-separable convolution, where the convolution kernel size of these two sub-convolutional layers is k×k and grouping convolution is performed. The fourth convolutional layer includes a convolutional layer and a convolutional kernel with a size of 1×1, or the fourth convolutional layer includes three sub-convolutional layers obtained by decomposing a convolutional layer with a size of k×k convolutional kernel using tensor decomposition and depth-unit separable convolution, and these three sub-convolutional layers may be a convolutional layer with a size of 1×k convolutional kernel and in which settling convolution is performed, a convolutional layer with a size of k×1 convolutional kernel and in which settling convolution is performed, and a convolutional layer with a size of 1×1 convolutional kernel.

[0072] In some embodiments, the residual units in the neural network filter may be separable for luminance and chromaticity; that is, the first residual unit in the neural network filter can be used to extract one image feature from the luminance and chromaticity components of the image awaiting filtering, and the neural network filter further includes a second residual unit, which is used to extract another image feature from the luminance and chromaticity components of the image awaiting filtering. In other words, the neural network filter includes residual units for the image luminance component and the image chromaticity component, respectively. In such cases, the residual block structure included in the residual unit for the image luminance component may or may not be the same as the residual block structure included in the residual unit for the image chromaticity component.

[0073] For example, the residual unit for the image luminance component (first residual unit) includes the first residual block in the above-described embodiment, and the residual unit for the image chromaticity component (second residual unit) includes at least one of the following: namely, a second residual block having the same structure as the first residual block in the above-described embodiment, and other residual blocks other than the first residual block. In some embodiments, the residual block included in the residual unit for the image luminance component (first residual unit) is the first residual block in the above-described embodiment, and the residual block included in the residual unit for the image chromaticity component (second residual unit) may be a residual block of a different structure, for example, the residual block shown in Figure 10 may be selected and used. The residual block shown in Figure 10 includes a convolutional layer with a convolutional kernel size of 1×1, a PReLU, a convolutional layer with a convolutional kernel size of 1×1, and a convolutional layer with a convolutional kernel size of 3×3. Of these, the size of the convolutional kernel shown in Figure 10 may be adjusted according to actual needs, and the activation function may be ReLU or GeLU.

[0074] In some embodiments, the neural network filter may further include a shallow feature extraction unit, which includes at least one convolutional layer, and the shallow feature extraction unit extracts shallow feature information from the input data and uses the shallow feature information to input to residual units, for example, the first residual unit and the second residual unit.

[0075] In some embodiments, the neural network filter may further include a feature mapping unit, which is used to perform mapping on the image feature information output from the residual unit. In some embodiments, when the residual unit in the neural network filter includes residual units for the image luminance component and the image chromaticity component, a first feature mapping unit and a second feature mapping unit can be provided for the image luminance component and the image chromaticity component, respectively. The first feature mapping unit for the image luminance component is connected to the first residual unit for the image luminance component, and the second feature mapping unit for the image chromaticity component is connected to the second residual unit for the image chromaticity component.

[0076] In step S730, a filtered image is obtained of the reconstructed image output from the neural network filter.

[0077] In some embodiments of this application, sample input data is generated during the training phase to train a neural network filter based on sample images (the sample input data is adjusted according to the application scenario of the neural network filter), and then the acquired sample input data is input to the neural network filter, and the parameters of the neural network filter are adjusted based on the loss value between the output of the neural network filter and the desired filtered result image corresponding to the sample image. This process is repeated until the neural network filter satisfies the convergence condition.

[0078] Furthermore, the technical proposal in the embodiment shown in Figure 7 can be applied to loop filtering in the video encoding and decoding process; that is, loop filtering can be performed in video encoding and decoding using the filtering method shown in Figure 7. Alternatively, the technical proposal in the embodiment shown in Figure 7 can also be applied to post-processing of video or images; that is, filtering can be performed on images decoded at the video playback end, images captured by terminal equipment, or other images.

[0079] Based on the neural network-based filtering method shown in Figure 7, the present invention further provides a video encoding method which can be executed by a device with computing capabilities, for example, by a terminal device or a server. The specific flow is shown in Figure 11 and includes the following steps S1110 to S1140.

[0080] In step S1110, input data for the neural network filter is acquired, and this input data includes at least a reconstructed image awaiting filtering.

[0081] For specific details of the implementation of this step, please refer to step S710 described above, and a detailed explanation will be omitted here.

[0082] In step S1120, input data is input to a neural network filter, which includes a first residual unit for extracting image feature information. This first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among these plurality of residual blocks is arranged in parallel and includes a plurality of convolutional layers with different convolutional kernel sizes.

[0083] For specific details of the implementation of this step, please refer to step S720 described above, and a detailed explanation will be omitted here.

[0084] In step S1130, a filtered image is obtained of the reconstructed image output from the neural network filter.

[0085] For specific details of the implementation of this step, please refer to step S730 described above, and a detailed explanation will be omitted here.

[0086] In step S1140, a predicted image corresponding to the next frame is generated based on the filtered image, and the next frame is encoded based on the predicted image corresponding to the next frame.

[0087] In some embodiments, after obtaining a filtered image, the flow shown in Figure 3 can be referenced. Specifically, the filtered image of the reconstructed image is used as a reference image for the next frame to perform motion estimation and motion compensation prediction. Subsequently, a predicted image for the next frame is obtained based on the motion compensation prediction result and the in-frame prediction result. The flow shown in Figure 3 is repeated until video image encoding is complete.

[0088] Accordingly, based on the neural network-based filtering method shown in Figure 7, the present invention further provides a video decoding method, which may be executed by a device with computing processing capabilities, for example, by a terminal device or a server. The specific flow is shown in Figure 12 and includes the following steps S1210 to S1240.

[0089] In step S1210, input data for the neural network filter is acquired, and this input data includes at least a reconstructed image awaiting filtering.

[0090] For specific details of the implementation of this step, please refer to step S710 described above, and a detailed explanation will be omitted here.

[0091] In step S1220, input data is input to a neural network filter, which includes a first residual unit for extracting image feature information. This first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among these plurality of residual blocks is arranged in parallel and includes a plurality of convolutional layers with different convolutional kernel sizes.

[0092] For specific details of the implementation of this step, please refer to step S720 described above, and a detailed explanation will be omitted here.

[0093] In step S1230, a filtered image is obtained of the reconstructed image output from the neural network filter.

[0094] For specific details of the implementation of this step, please refer to step S730 described above, and a detailed explanation will be omitted here.

[0095] In step S1240, a predicted image corresponding to the next frame is generated based on the filtered image, and a decoding process is performed on the video bitstream based on the predicted image corresponding to the next frame.

[0096] In several embodiments, after obtaining a filtered image of the reconstructed image, motion estimation and motion compensation prediction are performed using the filtered image of the reconstructed image as a reference image for the next frame. Subsequently, a predicted image of the next frame is obtained based on the motion compensation prediction results and the in-frame prediction results. Then, the reconstructed image of the next frame is generated by superimposing the predicted image of the next frame with the reconstructed residual signal that has undergone inverse quantization and inverse transformation. By repeating this process, decoding of the video bitstream can be achieved.

[0097] The following describes further details of the embodiments of this application that are not described in detail in conjunction with Figures 13 to 17.

[0098] In the embodiments of this application, a neural network-based filtering scheme is proposed, which can strike a good balance between encoding performance and computational complexity by designing a novel neural network filter structure using methods such as luminance and chromaticity separation, multi-stage receptive fields, and CPD. The scheme can be applied to video encoders and decoders, or to video pre-processing and post-processing products.

[0099] In some embodiments, Figure 13 shows the structure of a neural network filter in an embodiment of the present application, which includes an input section, a head section, an intermediate section, a tail section, and an output section.

[0100] In some embodiments, the input to a neural network filter may include two parts: image information and edge information. The image information includes a reconstruction image awaiting filtering, a prediction image corresponding to the reconstruction image awaiting filtering, and boundary strength, thereby providing the main image content for filtering the encoded and decoded reconstruction image. The edge information includes sliceQP, baseQP, and IPB. Of these, sliceQP and baseQP are used to represent the quantization parameters and distortion level of the reconstruction image awaiting filtering, and IPB is used to represent the encoding type of the reconstruction image awaiting filtering (i.e., whether it is an I-frame, P-frame, or B-frame).

[0101] The input to the neural network filter includes, but is not limited to, the information described above. Furthermore, the reconstructed image awaiting filtering at the input to the neural network filter is essential, while other information can be increased or decreased according to actual needs.

[0102] In some embodiments, the head of the neural network filter is used to extract shallow feature information from the input data and may include two 1x3 convolutional layers, two 3x1 convolutional layers, and one PReLU activation layer. The input data is concatenated and sent to the head module of the neural network filter for feature extraction, thereby obtaining a shallow feature representation.

[0103] In some embodiments, the intermediate section of the neural network filter is a crucial backbone structure of the network and may include multiple residual blocks (ResBlock). In some embodiments, the intermediate section may employ a form of luminance and chromaticity separation, that is, different modules, Y-Part and UV-Part, may be used to process the luminance and chromaticity components, respectively.

[0104] In some embodiments of this application, the residual block structure, as shown in Figure 14, can obtain multi-stage receptive field information by using 3×3 convolutions and 1×1 convolutions in a jointed manner.

[0105] In some embodiments of this application, the structure of the residual block can be decomposed from one 3×3 convolution to one 3×1 convolution and one 1×3 convolution according to the principle of CPD, as shown in Figure 15.

[0106] In some embodiments of this application, the structure of the residual block, as shown in Figure 16, allows a single 3x3 convolution to be decomposed into a single 3x3 convolution performing a settling convolution and a single 1x1 convolution using a depth-separable convolutional DSC. Of these, g indicates the use of a settling convolution, i.e., it specifies that the convolutions within the channel range use the same parameters.

[0107] In some embodiments of this application, the structure of the residual block, as shown in Figure 17, allows for the decomposition of one 3×3 convolution into one 1×3 convolution performing a settling convolution, one 3×1 convolution performing a settling convolution, and one 1×1 convolution using CPD and DSC simultaneously. Of these, g indicates the use of a settling convolution, i.e., it specifies that the convolutions within the channel range use the same parameters.

[0108] Furthermore, each residual block included in the Y-Part and UV-Part in the intermediate section shown in Figure 13 may adopt any one of the structures shown in Figures 14 to 17, or some of the residual blocks may adopt any one of the structures shown in Figures 14 to 17. For example, the Y-Part contains four residual blocks, and all four of these residual blocks may adopt the structure shown in Figure 14, or two of them may adopt the structure shown in Figure 15 and the other two may adopt the structure shown in Figure 16.

[0109] In some embodiments, the number and / or structure of residual blocks included in the Y-Part and UV-Part may be the same or different. For example, different residual blocks may be used in the Y-Part and UV-Part in the intermediate section. For instance, one or more residual blocks shown in Figures 14 to 17 may be used in the Y-Part, while other residual blocks, such as those shown in Figure 10, may be used in the UV-Part.

[0110] In some embodiments, referring again to Figure 13, as shown in Figure 13, the tail network of the neural network filter can also adopt a form of luminance and chromaticity separation to correspond to the separation of luminance and chromaticity in the intermediate region, thereby obtaining a luminance-filtered image and a chromaticity-filtered image, respectively. In some embodiments, the tail may include three convolutional layers and one PReLU activation layer, thereby enabling the mapping of feature information with a relatively high number of channels to image information with a relatively low number of channels.

[0111] In some embodiments, the final output of the neural network filter, i.e., the filtered image, can be obtained by adding the reconstructed image awaiting filtering, which is input to the neural network filter, with the residuals learned by the network. That is, in the neural network filter structure shown in Figure 13, the neural network is used to learn residual data, and in this way, because the amount of residual data is relatively small, the processing efficiency of the neural network can be increased. In other embodiments of this application, the output of the network at the tail of the neural network filter may be used directly as the output of the neural network filter, and in this way, the neural network filter learns the difference between the images before and after filtering.

[0112] The residual block structure and convolution kernel size in the embodiments of this application are merely illustrative and may be adjusted according to the actual application scenario in other embodiments of this application.

[0113] The following describes embodiments of the apparatus of this application, which may be used to carry out the methods described in the embodiments of this application described above. For details not described in the embodiments of the apparatus of this application, refer to the embodiments of the methods described above.

[0114] Figure 18 is a block diagram of a neural network-based filtering device in several embodiments of the present application, and this neural network-based filtering device can be applied to a device with computing capabilities, such as a terminal device or a server.

[0115] As shown in Figure 18, the neural network-based filtering device 1800 in some embodiments of this application includes a generation unit 1802, a processing unit 1804, and an acquisition unit 1806.

[0116] Of these, the generation unit 1802 is configured as follows: it acquires input data for a neural network filter, and the input data includes at least the images awaiting filtering; the processing unit 1804 is configured as follows: it inputs the input data to the neural network filter, and the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and with different convolutional kernel sizes; and the acquisition unit 1806 is configured as follows: it acquires filtered images of the images awaiting filtering output from the neural network filter.

[0117] For specific functions and structures of each of the above-mentioned units, please refer to the previously described embodiments; a detailed explanation is omitted here.

[0118] Figure 19 is a block diagram of a video encoding device in several embodiments of the present application, which can be used in a device with computing capabilities, such as a terminal device or a server.

[0119] As shown in Figure 19, the video encoding apparatus 1900 in some embodiments of this application includes a generation unit 1902, a processing unit 1904, an acquisition unit 1906, and an encoding unit 1908.

[0120] Of these, the generation unit 1902 is configured to acquire input data for a neural network filter, and the input data includes at least a reconstructed image awaiting filtering; the processing unit 1904 is configured to input the input data to the neural network filter, and the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, the first residual block of the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and with different convolutional kernel sizes; the acquisition unit 1906 is configured to acquire a filtered image of the reconstructed image output from the neural network filter; and the encoding unit 1908 is configured as follows: it generates a predicted image corresponding to the image of the next frame based on the filtered image, and performs encoding processing on the image of the next frame based on the predicted image corresponding to the image of the next frame.

[0121] Figure 20 is a block diagram of a video decoding device in several embodiments of the present application, and the video decoding device can be used in equipment having computing processing functions, such as terminal equipment or servers.

[0122] As shown in Figure 20, the video decoding apparatus 2000 in some embodiments of this application includes a generation unit 2002, a processing unit 2004, an acquisition unit 2006, and a decoding unit 2008.

[0123] Of these, the generation unit 2002 is configured to acquire input data for a neural network filter, and the input data includes at least a reconstructed image awaiting filtering; the processing unit 2004 is configured to input the input data to the neural network filter, and the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, the first residual block of the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and with different convolutional kernel sizes; the acquisition unit 2006 is configured to acquire a filtered image of the reconstructed image output from the neural network filter; and the decoding unit 2008 is configured as follows: it generates a predicted image corresponding to the image of the next frame based on the filtered image, and performs decoding processing on the video bitstream based on the predicted image corresponding to the image of the next frame.

[0124] Figure 21 shows the structure of a computer system for realizing an electronic device according to an embodiment of this application.

[0125] Note that the electronic computer system 2100 shown in Figure 21 is merely one example and does not limit the functions and scope of application of the embodiments of this application.

[0126] As shown in Figure 21, the computer system 2100 includes a Central Processing Unit (CPU) 2101, which can perform various appropriate operations and processes based on programs stored in the ROM (Read-Only Memory) 2102 or programs loaded from the storage unit 2108 into the RAM (Random Access Memory) 2103, for example, it can perform the methods described in the embodiments described above. The RAM 2103 further stores various programs and data necessary for the operation of the system. The CPU 2101, ROM 2102, and RAM 2103 are connected to each other via a bus 2104. An Input / Output (I / O) interface 2105 is also connected to the bus 2104.

[0127] Furthermore, the following components may be connected to the I / O interface 2105: namely, an input unit 2106 including a keyboard and mouse; an output unit 2107 including a CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), and speakers; a storage unit 2108 including a hard disk; and a communication unit 2109 including a LAN (Local Area Network) card and a network interface card such as a modem. The communication unit 2109 performs communication processing via a network such as the Internet. A drive 2110 may also be connected to the I / O interface 2105 as needed. Removable media 2111, such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memory, can be set in the drive 2110 as needed, and computer programs read from them can be installed in the storage unit 2108 according to the needs.

[0128] In particular, according to embodiments of this application, the process described with reference to the flowchart above can be implemented as a computer software program. For example, embodiments of this application include a computer program product which includes a computer program carried on a computer-readable medium, and the computer program is used to perform the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication unit 2109 and / or installed from a removable medium 2111. When executed by a central processing unit (CPU) 2101, the computer program can perform various functions limited to the system of this application.

[0129] In another aspect, the present application further provides a computer-readable medium which may be included in the electronic device described in the above embodiments, or which may exist independently without being provided in the electronic device. The computer-readable medium cancels one or more computer programs which, when executed by the electronic device, can enable the electronic device to implement the methods described in the above embodiments.

[0130] Although the detailed description above refers to multiple modules or units of the device for performing operations, such division is not mandatory. In practice, according to embodiments of this application, the features and functions of the above-mentioned multiple modules or units can be integrated and implemented in a single module or unit. Alternatively, the features and functions of the above-mentioned single module or unit can be further divided and implemented in multiple modules or units.

[0131] Furthermore, through the description of the embodiments described above, those skilled in the art will understand that the exemplary embodiments described herein may be implemented by software, or by a combination of software and necessary hardware. Thus, the technical solutions in the embodiments of this application can be implemented in the form of a software product, which can be stored on a non-volatile storage medium (e.g., D-ROM, USB, mobile hard disk, etc.) or a network, and which can include a number of instructions, thereby causing an electronic device (e.g., a personal computer, server, touch terminal, network device, etc.) to execute the method in the embodiments of this application.

[0132] While preferred embodiments of this application have been described above, this application is not limited to these embodiments, and any modifications to this application that do not deviate from the spirit of this application fall within the technical scope of this application.

Claims

1. A method for performing filtering based on a neural network filter, which is executed by a device having computational processing capabilities, A step of acquiring input data for a neural network filter, wherein the input data includes at least an image awaiting filtering; A step of inputting the aforementioned input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and having different convolutional kernel sizes; and A method comprising the step of obtaining filtered images of the images awaiting filtering, which are output from the neural network filter.

2. The method according to claim 1, The method further includes at least one of the following in the input data: boundary intensity information, slice quantization parameters, basic quantization parameters, frame type of the image awaiting filtering, and a corresponding predicted image when the image awaiting filtering is a reconstructed image.

3. The method according to claim 1, The plurality of convolutional layers include a first convolutional layer and a second convolutional layer, and the first residual block further includes a first activation function layer connected to the first convolutional layer, a second activation function layer connected to the second convolutional layer, and a third and fourth convolutional layer connected sequentially. A method wherein the input terminal of the first convolutional layer is connected to the input terminal of the second convolutional layer and serves as the input terminal of the first residual block, the output terminal of the first activation function layer is connected to the output terminal of the second activation function layer and serves as the input terminal of the third convolutional layer, and the superposition of the input data of the first residual block and the output data of the fourth convolutional layer serves as the output data of the first residual block.

4. The method according to claim 3, A method wherein the size of the convolution kernel of the first convolutional layer is n × n, and the size of the convolution kernel of the second convolutional layer is m × m, where m and n are positive integers and m ≠ n.

5. The method according to claim 3, The first convolutional layer includes two sub-convolutional layers obtained by decomposing a convolutional layer with a convolutional kernel size of n × n by tensor decomposition, and the convolutional kernel size of the second convolutional layer is m × m, where m and n are positive integers and m ≠ n.

6. The method according to claim 5, The method comprises a first subconvolutional layer having a convolutional kernel size of 1 × n and a second subconvolutional layer having a convolutional kernel size of n × 1.

7. The method according to claim 3, The first convolutional layer comprises two sub-convolutional layers obtained by decomposing a convolutional layer having a convolutional kernel size of n × n by depth-separable convolution, the size of the convolutional kernel of the second convolutional layer is m × m, where m and n are positive integers and m ≠ n.

8. The method according to claim 7, The method comprises a third subconvolutional layer in which a convolution kernel of size n × n and a settling convolution is performed, and a fourth subconvolutional layer in which a convolution kernel of size 1 × 1.

9. The method according to claim 3, The first convolutional layer comprises three sub-convolutional layers obtained by decomposing a convolutional layer with a convolutional kernel size of n × n by tensor decomposition and depth-separable convolution, the second convolutional layer has a convolutional kernel size of m × m, m and n are positive integers, and m ≠ n, in this method.

10. The method according to claim 9, A method comprising the three subconvolutional layers, a fifth subconvolutional layer having a convolutional kernel size of 1 × n and performing a settling convolution, a sixth subconvolutional layer having a convolutional kernel size of n × 1 and performing a settling convolution, and a seventh subconvolutional layer having a convolutional kernel size of 1 × 1.

11. The method according to claim 3, The aforementioned fourth convolutional layer is A single convolutional layer with a convolutional kernel size of k × k; Two sub-convolutional layers obtained by decomposing a convolutional layer with a convolution kernel size of k×k using tensor decomposition; Two sub-convolutional layers obtained by decomposing a convolutional layer with a convolutional kernel size of k×k by depth-separable convolution; and Three sub-convolutional layers obtained by decomposing a convolutional layer with a kernel size of k×k through tensor decomposition and depth-separable convolution. It includes one of the following: Here, k is a positive integer, method.

12. The method according to claim 1, The first residual unit is used to extract one image feature information from the luminance component and chromaticity component of the image awaiting filtering, and the neural network filter further includes a second residual unit, the second residual unit is used to extract another image feature information from the luminance component and chromaticity component of the image awaiting filtering. The second residual unit includes a plurality of sequentially connected residual blocks, wherein the structure of the plurality of residual blocks is the same as or different from the structure of the residual blocks included in the first residual unit.

13. The method according to claim 12, The first residual unit is used to extract image feature information of the luminance component of the image awaiting filtering, and the second residual unit is used to extract image feature information of the chromaticity component of the image awaiting filtering. A method wherein the second residual unit includes at least one of a second residual block having the same structure as the first residual block, and other residual blocks other than the second residual block.

14. The method according to claim 12, The neural network filter further includes a shallow feature extraction unit, A method wherein the shallow feature extraction unit includes at least one convolutional layer, and the shallow feature extraction unit is used to extract shallow feature information from the input data and to input the shallow feature information to the first residual unit and the second residual unit.

15. The method according to claim 12, The aforementioned neural network filter is A first feature mapping unit for performing mapping processing on image feature information output from the first residual unit; and The system further includes a second feature mapping unit for performing mapping processing on image feature information output from the second residual unit, The first feature mapping unit is connected to the first residual unit, and the second feature mapping unit is connected to the second residual unit, in this method.

16. A method for encoding video, performed by a device having a computing function, A step of acquiring input data for a neural network filter, wherein the input data includes at least a reconstructed image awaiting filtering; A step of inputting the aforementioned input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and having different convolutional kernel sizes; A step of obtaining a filtered image of the reconstructed image output from the neural network filter; and A method comprising the steps of generating a predicted image corresponding to the image of the next frame based on the filtered image, and performing an encoding process on the image of the next frame based on the predicted image corresponding to the image of the next frame.

17. A method for decoding video, performed by a device having a computing function, A step of acquiring input data for a neural network filter, wherein the input data includes at least a reconstructed image awaiting filtering; A step of inputting the aforementioned input data into the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and having different convolutional kernel sizes; A step of obtaining a filtered image of the reconstructed image output from the neural network filter; and A method comprising the steps of generating a predicted image corresponding to the image of the next frame based on the filtered image, and performing a decoding process on a video bitstream based on the predicted image corresponding to the image of the next frame.

18. A filtering device based on a neural network, A generation unit that acquires input data for a neural network filter, wherein the input data includes at least an image awaiting filtering; A processing unit that inputs the aforementioned input data to the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and having different convolutional kernel sizes; and A filtering device including an acquisition unit that acquires filtered images of the images awaiting filtering, which are output from the neural network filter.

19. A video encoding device, A generation unit that acquires input data for a neural network filter, wherein the input data includes at least a reconstructed image awaiting filtering; A processing unit that inputs the aforementioned input data to the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and having different convolutional kernel sizes; An acquisition unit that acquires a filtered image of the reconstructed image output from the neural network filter; and A video encoding apparatus including an encoding unit that generates a predicted image corresponding to the image of the next frame based on the filtered image, and performs encoding processing on the image of the next frame based on the predicted image corresponding to the image of the next frame.

20. A video decoding device, A generation unit that acquires input data for a neural network filter, wherein the input data includes at least a reconstructed image awaiting filtering; A processing unit that inputs the aforementioned input data to the neural network filter, wherein the neural network filter includes a first residual unit for extracting image feature information, the first residual unit includes a plurality of sequentially connected residual blocks, and the first residual block among the plurality of residual blocks includes a plurality of convolutional layers arranged in parallel and having different convolutional kernel sizes; An acquisition unit that acquires a filtered image of the reconstructed image output from the neural network filter; and A video decoding apparatus including a decoding unit that generates a predicted image corresponding to the image of the next frame based on the filtered image, and performs decoding processing on a video bitstream based on the predicted image corresponding to the image of the next frame.

21. It is an electronic device, Processor; and Includes a memory connected to the aforementioned processor, The memory device stores a computer program. The processing device is configured to implement the method described in any one of claims 1 to 17 by executing a computer program, an electronic device.

22. A program for causing a computer to perform the method described in any one of claims 1 to 17.

Images