Filtering and encoding and decoding method, device, computer readable medium and electronic device

Abstract

The embodiments of the present application provide a filtering and encoding and decoding method, device, computer-readable medium and electronic device. The filtering method based on deep learning includes: obtaining a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; generating input parameters of a deep learning filter according to the luminance component reconstructed image and the chrominance component information; inputting the input parameters into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter. The technical solution of the embodiments of the present application can improve the filtering effect, thereby helping to improve the encoding and decoding efficiency of the video.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular to a filtering and encoding and decoding method, device, computer-readable medium and electronic device. Background Art

[0002] In the field of video coding and decoding, after the predicted image and the reconstructed residual image are superimposed to generate a reconstructed image, the reconstructed image will be distorted. Therefore, in order to obtain a higher quality image, it is usually necessary to perform loop filtering on the reconstructed image. In the loop filtering process, how to improve the filtering effect to improve the coding and decoding efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a filtering and encoding and decoding method, device, computer-readable medium and electronic device, which can improve the filtering effect at least to a certain extent, thereby helping to improve the encoding and decoding efficiency of the video.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a filtering method based on deep learning is provided, including: obtaining a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; generating input parameters of a deep learning filter according to the luminance component reconstructed image and the chrominance component information; inputting the input parameters into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter.

[0006] According to one aspect of an embodiment of the present application, a video encoding method is provided, including: obtaining a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; generating input parameters of a deep learning filter according to the luminance component reconstructed image and the chrominance component information; inputting the input parameters into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter; generating a luminance component predicted image corresponding to a next frame of image based on the filtered image, and encoding the next frame of video image based on the luminance component predicted image.

[0007] According to one aspect of an embodiment of the present application, a video decoding method is provided, including: obtaining a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; generating input parameters of a deep learning filter according to the luminance component reconstructed image and the chrominance component information; inputting the input parameters into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter; generating a luminance component predicted image corresponding to the next frame image based on the filtered image, and decoding a video code stream based on the luminance component predicted image.

[0008] According to one aspect of an embodiment of the present application, a deep learning-based filtering device is provided, comprising: an acquisition unit, configured to acquire a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; a generation unit, configured to generate input parameters of a deep learning filter according to the luminance component reconstructed image and the chrominance component information; a processing unit, configured to input the input parameters into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the chrominance component information includes at least one of the following: a chrominance component pre-filtering image corresponding to the encoded image, a chrominance component prediction image corresponding to the encoded image, and chrominance component block division information corresponding to the encoded image.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the generation unit is configured to: upsample the chromaticity component information to obtain chromaticity component information of the same size as the luminance component reconstructed image; perform layer merging processing on the luminance component reconstructed image and the upsampled chromaticity component information, and use the result of the layer merging processing as the input parameter of the deep learning filter.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the generation unit is configured to: upsample the chrominance component information to obtain chrominance component information of the same size as the luminance component reconstructed image, the chrominance component information including: a chrominance component pre-filtering image, a chrominance component prediction image, and chrominance component block division information; merge the luminance component reconstructed image with the chrominance component pre-filtering image after upsampling, and extract the features of the merged image to obtain a first feature; merge the luminance component prediction image corresponding to the encoded image with the chrominance component prediction image after upsampling, and extract the features of the merged image to obtain a second feature; merge the luminance component block division information corresponding to the encoded image with the chrominance component block division information after upsampling, and extract the features of the merged image to obtain a third feature; generate the input parameters according to the first feature, the second feature, and the third feature.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the deep learning filter includes a convolution unit, a residual unit and a rearrangement unit connected in sequence, and the residual unit includes at least one residual block.

[0013] In some embodiments of the present application, based on the aforementioned scheme, if the residual unit includes multiple residual blocks, the number of channels of the multiple residual blocks is the same, or the number of channels of each residual block in the multiple residual blocks is not completely the same.

[0014] In some embodiments of the present application, based on the aforementioned scheme, a residual block includes: a first convolutional layer, a parameterized rectified linear unit, and a second convolutional layer connected in sequence; wherein the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the second convolutional layer is used as the output of the residual block.

[0015] In some embodiments of the present application, based on the aforementioned scheme, a residual block includes: a first convolutional layer, a parameterized rectified linear unit, a second convolutional layer and a convolutional block attention unit connected in sequence; wherein, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the convolutional block attention unit is used as the output of the residual block.

[0016] In some embodiments of the present application, based on the aforementioned scheme, a residual block includes: a first convolutional layer, a parameterized rectified linear unit, a third convolutional layer, and a second convolutional layer connected in sequence; wherein the convolution kernel of the first convolutional layer is the same size as the convolution kernel of the third convolutional layer, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the second convolutional layer is used as the output of the residual block.

[0017] In some embodiments of the present application, based on the aforementioned scheme, a residual block includes: a first convolutional layer, a parameterized rectified linear unit, a third convolutional layer, a second convolutional layer and a convolutional block attention unit connected in sequence; wherein the convolution kernel of the first convolutional layer is the same size as the convolution kernel of the third convolutional layer, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the convolutional block attention unit is used as the output of the residual block.

[0018] According to one aspect of an embodiment of the present application, a video encoding device is provided, comprising: an acquisition unit, configured to acquire a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; a generation unit, configured to generate input parameters of a deep learning filter based on the luminance component reconstructed image and the chrominance component information; a processing unit, configured to input the input parameters to the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter; an encoding unit, configured to generate a luminance component prediction image corresponding to a next frame of image based on the filtered image, and to encode the next frame of video image based on the luminance component prediction image.

[0019] According to one aspect of an embodiment of the present application, a video decoding device is provided, comprising: an acquisition unit, configured to acquire a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; a generation unit, configured to generate input parameters of a deep learning filter according to the luminance component reconstructed image and the chrominance component information; a processing unit, configured to input the input parameters into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter; a decoding unit, configured to generate a luminance component predicted image corresponding to a next frame image based on the filtered image, and to decode a video code stream based on the luminance component predicted image.

[0020] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the filtering method, video encoding method or video decoding method based on deep learning as described in the above embodiments is implemented.

[0021] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements a deep learning-based filtering method, a video encoding method or a video decoding method as described in the above embodiments.

[0022] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the filtering method based on deep learning, the video encoding method, or the video decoding method provided in the above-mentioned various optional embodiments.

[0023] In the technical solutions provided in some embodiments of the present application, by obtaining a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image, input parameters of a deep learning filter are generated according to the luminance component reconstructed image and the chrominance component information, and then the input parameters are input into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter, so that when filtering the luminance component of the image, the chrominance component information can be fully utilized, and then the existing chrominance component information can be used to improve the performance of the deep learning filter for the luminance component, thereby improving the filtering effect, which is beneficial to improving the encoding and decoding efficiency of the video.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0026] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;

[0027] Figure 2 A schematic diagram showing the placement of a video encoding device and a video decoding device in a streaming transmission system;

[0028] Figure 3 shows a basic flow chart of a video encoder;

[0029] Figure 4 A schematic diagram of filtering processing based on CNNLF is shown;

[0030] Figure 5 A schematic diagram showing filtering of luminance component and chrominance component;

[0031] Figure 6 A schematic diagram showing filtering of a luminance component according to an embodiment of the present application is shown;

[0032] Figure 7 A flowchart of a filtering method based on deep learning according to an embodiment of the present application is shown;

[0033] Fig. 8A A schematic diagram showing a method of generating input parameters of a deep learning filter based on a luminance component reconstructed image and chrominance component information according to an embodiment of the present application;

[0034] Figure 8B A schematic diagram showing a method of generating input parameters of a deep learning filter based on a luminance component reconstructed image and chrominance component information according to an embodiment of the present application;

[0035] Fig. 9 A schematic diagram showing a method of generating input parameters of a deep learning filter based on a luminance component reconstructed image and chrominance component information according to an embodiment of the present application;

[0036] Fig.10 A schematic diagram of the structure of a residual block according to an embodiment of the present application is shown;

[0037] Fig.11 A schematic diagram of the structure of a residual block according to an embodiment of the present application is shown;

[0038] Fig.12 A schematic diagram of the structure of a residual block according to an embodiment of the present application is shown;

[0039] Fig.13 A schematic diagram of the structure of a residual block according to an embodiment of the present application is shown;

[0040] Fig.14 A flowchart of a video encoding method according to an embodiment of the present application is shown;

[0041] Fig.15 A flowchart of a video decoding method according to an embodiment of the present application is shown;

[0042] Fig.16 A block diagram of a filtering device based on deep learning according to an embodiment of the present application is shown;

[0043] Fig.17 A block diagram of a video encoding device according to an embodiment of the present application is shown;

[0044] Fig.18 A block diagram of a video decoding device according to an embodiment of the present application is shown;

[0045] Fig.19 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] The exemplary embodiments are now described in a more comprehensive manner with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be understood as being limited to these examples; on the contrary, the purpose of providing these embodiments is to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0047] In addition, the features, structures or characteristics described in the present application may be combined in one or more embodiments in any suitable manner. In the following description, there are many specific details so that the embodiments of the present application can be fully understood. However, those skilled in the art will appreciate that when implementing the technical scheme of the present application, all the detailed features in the embodiments may not be needed, one or more specific details may be omitted, or other methods, elements, devices, steps, etc. may be adopted.

[0048] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0049] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0050] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0051] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0052] like Figure 1As shown, the system architecture 100 includes a plurality of terminal devices, which can communicate with each other through, for example, a network 150. For example, the system architecture 100 may include a first terminal device 110 and a second terminal device 120 interconnected through the network 150. Figure 1 In the embodiment of the present invention, the first terminal device 110 and the second terminal device 120 perform unidirectional data transmission.

[0053] For example, the first terminal device 110 may encode video data (e.g., a video picture stream captured by the terminal device 110) for transmission to the second terminal device 120 via the network 150. The encoded video data is transmitted in the form of one or more encoded video code streams. The second terminal device 120 may receive the encoded video data from the network 150, decode the encoded video data to restore the video data, and display the video picture based on the restored video data.

[0054] In one embodiment of the present application, the system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 that perform bidirectional transmission of encoded video data, which may occur during a video conference, for example. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 may encode video data (e.g., a video picture stream collected by the terminal device) to transmit to the other terminal device of the third terminal device 130 and the fourth terminal device 140 through the network 150. Each of the third terminal device 130 and the fourth terminal device 140 may also receive the encoded video data transmitted by the other terminal device of the third terminal device 130 and the fourth terminal device 140, and may decode the encoded video data to restore the video data, and may display the video picture on an accessible display device according to the restored video data.

[0055] exist Figure 1 In the embodiment of the present invention, the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140 may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network 150 represents any number of networks that transmit encoded video data between the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140, including, for example, wired and / or wireless communication networks. The communication network 150 may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purpose of the present application, unless explained below, the architecture and topology of the network 150 may be irrelevant to the operation disclosed in the present application.

[0056] In one embodiment of the present application, Figure 2 The placement of the video encoding device and the video decoding device in the streaming environment is shown. The subject matter disclosed in the present application can be equally applicable to other video-supported applications, including, for example, video conferencing, digital TV (television), storing compressed video on digital media including CD, DVD, memory stick, etc.

[0057] The streaming system may include an acquisition subsystem 213, which may include a video source 201 such as a digital camera, which creates an uncompressed video picture stream 202. In an embodiment, the video picture stream 202 includes samples taken by a digital camera. The video picture stream 202 is depicted as a thick line to emphasize the high data volume of the video picture stream compared to the encoded video data 204 (or the encoded video code stream 204), and the video picture stream 202 can be processed by an electronic device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream 202, the encoded video data 204 (or the encoded video code stream 204) is depicted as a thin line to emphasize the lower data volume of the encoded video data 204 (or the encoded video code stream 204), which can be stored on the streaming server 205 for future use. One or more streaming client subsystems, such as Figure 2 , can access the streaming server 205 to retrieve the copies 207 and 209 of the encoded video data 204. The client subsystem 206 can include, for example, a video decoding device 210 in the electronic device 230. The video decoding device 210 decodes the incoming copy 207 of the encoded video data and generates an output video picture stream 211 that can be presented on a display 212 (e.g., a display screen) or another presentation device. In some streaming systems, the encoded video data 204, the video data 207, and the video data 209 (e.g., video bitstreams) can be encoded according to certain video encoding / compression standards.

[0058] It should be noted that the electronic device 220 and the electronic device 230 may include other components not shown in the figure. For example, the electronic device 220 may include a video decoding device, and the electronic device 230 may also include a video encoding device.

[0059] In one embodiment of the present application, taking the international video coding standards HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), and China's national video coding standard AVS as examples, after a video frame image is input, the video frame image will be divided into several non-overlapping processing units according to a block size, and each processing unit will perform similar compression operations. This processing unit is called CTU (Coding Tree Unit), or LCU (Largest Coding Unit). CTU can continue to be divided more finely to obtain one or more basic coding units CU (Coding Unit), which is the most basic element in a coding link.

[0060] The following introduces some concepts when encoding CU:

[0061] Predictive Coding: Predictive coding includes intra-frame prediction and inter-frame prediction. The original video signal is predicted by the selected reconstructed video signal to obtain the residual video signal. The encoder needs to decide which predictive coding mode to choose for the current CU and inform the decoder. Intra-frame prediction means that the predicted signal comes from the encoded and reconstructed area in the same image; inter-frame prediction means that the predicted signal comes from other images that have been encoded and different from the current image (called reference images).

[0062] Transform & Quantization: After the residual video signal is transformed by DFT (Discrete Fourier Transform), DCT (Discrete Cosine Transform), etc., the signal is converted into the transform domain, which is called the transform coefficient. The transform coefficient is further subjected to lossy quantization operation, which loses a certain amount of information, making the quantized signal conducive to compression expression. In some video coding standards, there may be more than one transform method to choose from, so the encoder also needs to select one of the transform methods for the current CU and inform the decoder. The degree of quantization is usually determined by the quantization parameter (QP). A larger QP value means that coefficients with a larger value range will be quantized to the same output, which usually results in greater distortion and lower bit rate; on the contrary, a smaller QP value means that coefficients with a smaller value range will be quantized to the same output, which usually results in less distortion and a higher bit rate.

[0063] Entropy Coding or Statistical Coding: The quantized transform domain signal will be statistically compressed and encoded according to the frequency of occurrence of each value, and finally output as a binary (0 or 1) compressed bit stream. At the same time, the encoding generates other information, such as the selected coding mode, motion vector data, etc., which also need to be entropy encoded to reduce the bit rate. Statistical coding is a lossless coding method that can effectively reduce the bit rate required to express the same signal. Common statistical coding methods include variable length coding (VLC) or context-based binary arithmetic coding (CABAC).

[0064] The CABAC process mainly includes three steps: binarization, context modeling, and binary arithmetic coding. After the input syntax elements are binarized, the binary data can be encoded through the conventional coding mode and the bypass coding mode. The bypass coding mode does not need to assign a specific probability model to each binary bit. The input binary bit bin value is directly encoded with a simple bypass encoder to speed up the entire encoding and decoding. In general, different syntax elements are not completely independent, and the same syntax element itself has a certain memory. Therefore, according to the conditional entropy theory, conditional encoding using other encoded syntax elements can further improve the encoding performance compared to independent encoding or memoryless encoding. These encoded symbol information used as conditions are called context. In the conventional coding mode, the binary bits of the syntax elements enter the context modeler sequentially. The encoder assigns a suitable probability model to each input binary bit according to the value of the previously encoded syntax element or binary bit. This process is called context modeling. The context model corresponding to the syntax element can be located through ctxIdxInc (context index increment) and ctxIdxStart (context index Start). After the bin value and the assigned probability model are sent to the binary arithmetic encoder for encoding, the context model needs to be updated according to the bin value, which is the adaptive process in encoding.

[0065] Loop Filtering: The changed and quantized signal will be reconstructed through inverse quantization, inverse transformation and prediction compensation operations to obtain a reconstructed image. Compared with the original image, due to the influence of quantization, some information of the reconstructed image is different from the original image, that is, the reconstructed image will produce distortion. Therefore, the reconstructed image can be filtered, such as deblocking filter (DB), SAO (Sample Adaptive Offset) or ALF (Adaptive Loop Filter) and other filters, which can effectively reduce the degree of distortion caused by quantization. Since these filtered reconstructed images will be used as a reference for subsequent encoded images to predict future image signals, the above filtering operation is also called loop filtering, that is, filtering operation within the encoding loop.

[0066] In one embodiment of the present application, Figure 3 The basic flow chart of a video encoder is shown in FIG. 1 , in which intra-frame prediction is used as an example for explanation. In which the original image signal s k [x,y] and predicted image signal Perform difference operation to obtain the residual signal u k [x,y], residual signal u k [x,y] is transformed and quantized to obtain the quantized coefficients. The quantized coefficients are entropy coded to obtain the encoded bit stream, and the reconstructed residual signal u' is obtained through inverse quantization and inverse transformation. k [x,y], predicted image signal and reconstructed residual signal u' k [x,y] superposition generates reconstructed image signal Reconstructing image signals On the one hand, it is input to the intra-frame mode decision module and the intra-frame prediction module for intra-frame prediction processing, and on the other hand, it is filtered through loop filtering and outputs the filtered image signal s' k [x,y], filtered image signal s' k [x,y] can be used as the reference image for the next frame for motion estimation and motion compensation prediction. Then based on the result of motion compensation prediction s' r [x+m x ,y+m y ] and intra prediction results Get the predicted image signal of the next frame And continue to repeat the above process until the encoding is completed.

[0067] The above loop filter can be implemented based on CNNLF (Convolutional Neural Network Loop Filter, a filter based on deep learning). The model structure of CNNLF includes basic modules such as convolution layer, activation function, fully connected layer and pooling layer. The model parameters need to be obtained through training. Figure 4 As shown in FIG. 1 , after the CNNLF is trained, the image before filtering can be input into the trained CNNLF, and finally the filtered image can be output.

[0068] For image filtering tasks, different filtering models can be trained separately for the luminance component (Y) and chrominance components (Cb and Cr) to improve the filtering performance for the luminance component and chrominance component. Figure 5 As shown in the method A, two deep learning filters can be used to filter the Y component and the {Cb, Cr} component respectively; or as shown in the method B, three deep learning filters can be used to filter the Y component, the Cb component and the Cr component respectively. Figure 5 Whether it is method A or method B shown in the figure, since the Y component contains more texture information, the information of the Y component is often introduced when filtering the chrominance components Cb and / or Cr to improve the classification accuracy of the filter, thereby improving the final filtering performance.

[0069] In order to improve the performance of the brightness component filter, the present application proposes to use the existing information to improve the performance of the deep learning filter for the brightness component. Figure 6 As shown, the information of the chrominance component {Cb, Cr} can be used to improve the performance of the deep learning filter for the luminance component Y. Optionally, the information of the chrominance component can be one or more of the image before the chrominance component is filtered (i.e., the chrominance component reconstructed image), the predicted image of the chrominance component, and the block division information of the chrominance component.

[0070] Among them, deep learning belongs to the category of artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0071] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0072] Machine Learning (ML) is a multi-disciplinary cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. The deep learning filter in the embodiment of the present application is a filter based on machine learning / deep learning.

[0073] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0074] Figure 7 A flowchart of a deep learning-based filtering method according to an embodiment of the present application is shown. The deep learning-based filtering method can be executed by a device with a computing and processing function, such as a terminal device or a server. Figure 7 As shown, the filtering method based on deep learning includes at least steps S710 to S730, which are described in detail as follows:

[0075] In step S710, a luminance component reconstructed image corresponding to the encoded image and chrominance component information corresponding to the encoded image are obtained.

[0076] In one embodiment of the present application, the reconstructed image corresponding to the encoded image is an image generated by superimposing the reconstructed residual image obtained after inverse quantization and inverse transformation and the predicted image, for example Figure 3 In the process shown, the reconstructed image is the predicted image signal and reconstructed residual signal u' k [x,y] Image signal generated by superposition The luminance component reconstructed image is the luminance part of the reconstructed image corresponding to the encoded image.

[0077] Optionally, the chrominance component information includes at least one of the following: a chrominance component pre-filtering image corresponding to the encoded image, a chrominance component prediction image corresponding to the encoded image, and chrominance component block division information corresponding to the encoded image.

[0078] Optionally, the chrominance component block division information may be an image generated according to the chrominance component block division result, for example, may include at least one of the following images: a binary image generated according to the chrominance component block division boundary; a binary image generated according to the filtering boundary of the chrominance component reconstructed image by a deblocking filter; a chrominance component block division mean image obtained by taking the average of the values ​​of all samples in the same block as the value of the block for the chrominance component reconstructed image before filtering.

[0079] In step S720, input parameters of the deep learning filter are generated according to the luminance component reconstructed image and the chrominance component information.

[0080] In one embodiment of the present application, the chrominance component information can be upsampled to obtain chrominance component information of the same size as the luminance component reconstructed image, and then the luminance component reconstructed image and the upsampled chrominance component information can be layer-merged, and the result of the layer-merging process can be used as the input parameter of the deep learning filter.

[0081] Optionally, the upsampling process can be implemented using an upsampling filter or through a deep learning module. The upsampling filter can be implemented based on, for example, Lanczos (an algorithm that transforms a symmetric matrix into a symmetric tridiagonal matrix through an orthogonal similarity transformation), nearest (nearest neighbor interpolation algorithm), bilinear (bilinear interpolation algorithm), bicubic (bicubic interpolation algorithm) and other algorithms.

[0082] In one embodiment of the present application, the chrominance component information after upsampling processing may be a chrominance component reconstructed image, such as Fig. 8A As shown, the reconstructed image of the luminance component (ie, the Y component) and the reconstructed image of the chrominance component after upsampling (ie, the Cb and Cr components) can be merged as a layer respectively, and then input into the deep learning filter to finally obtain the image Y' after the luminance component filtering.

[0083] Optionally, the deep learning filter includes a convolution unit, a residual unit and a rearrangement unit connected in sequence. The convolution unit includes a convolutional layer (Convolutional layer, referred to as conv) and a parametric rectified linear unit (Parametric Rectified Linear Unit, referred to as prelu); the residual unit includes N residual blocks connected in sequence; the rearrangement unit is a shuffle unit. The shuffle unit here implements image upsampling processing by rearrangement. Fig. 8A In the embodiment shown in , the step size of the convolution layer in the convolution unit before the residual unit is s=2; another convolution layer may be provided between the residual unit and the shuffle unit, and the step size of the convolution layer is s=1.

[0084] In one embodiment of the present application, the chrominance component information after upsampling processing may be a chrominance component reconstructed image, a chrominance component predicted image, and chrominance component block division information, such as Figure 8B As shown, the luminance component reconstructed image, luminance component predicted image and luminance component block division information (i.e., Y component) and the upsampled chrominance component reconstructed image, chrominance component predicted image and chrominance component block division information (i.e., Cb and Cr components) can be merged as a layer respectively, and then input into the deep learning filter to finally obtain the luminance component filtered image Y'.

[0085] Optionally, the deep learning filter includes a convolution unit, a residual unit and a shuffle unit connected in sequence. The convolution unit includes a convolution layer conv and a parameterized rectified linear unit prelu; the residual unit includes N residual blocks connected in sequence; and the shuffle unit is a shuffle unit. Figure 8B In the embodiment shown in , the step size of the convolution layer in the convolution unit before the residual unit is s=2; another convolution layer may be provided between the residual unit and the shuffle unit, and the step size of the convolution layer is s=1.

[0086] It should be noted that, similar to the chrominance component block division information, the luminance component block division information in the embodiment of the present application may be an image generated based on the luminance component block division result, and may include, for example, at least one of the following images: a binary image generated based on the luminance component block division boundary; a binary image generated based on the filtering boundary of the luminance component reconstructed image by a deblocking filter; and a luminance component block division mean image obtained by taking the average of all sample points in the same block as the value of the block for the luminance component reconstructed image before filtering.

[0087] In one embodiment of the present application, the chrominance component information can be upsampled to obtain chrominance component information of the same size as the luminance component reconstructed image, the chrominance component information including: a chrominance component pre-filtered image, a chrominance component predicted image, and chrominance component block division information; then the luminance component reconstructed image and the chrominance component pre-filtered image after upsampling are merged, and the merged image features are extracted to obtain a first feature; the luminance component predicted image corresponding to the encoded image and the chrominance component predicted image after upsampling are merged, and the merged image features are extracted to obtain a second feature; the luminance component block division information corresponding to the encoded image and the chrominance component block division information after upsampling are merged, and the merged image features are extracted to obtain a third feature; and then the input parameters of the deep learning filter are generated according to the first feature, the second feature, and the third feature.

[0088] Specifically, Fig. 9 As shown, the luminance component prediction image is merged with the upsampled chrominance component prediction image and then passes through a convolution unit (the convolution unit includes a convolution layer conv and a parameterized rectified linear unit prelu, and the step size of the convolution layer is s=1) to extract features; the luminance component block division information is merged with the upsampled chrominance component block division information and then passes through a convolution unit (the convolution unit includes a convolution layer conv and a parameterized rectified linear unit prelu, and the step size of the convolution layer is s=1) to extract features; the luminance component reconstructed image is merged with the upsampled chrominance component reconstructed image and then passes through a convolution unit (the convolution unit includes a convolution layer conv and a parameterized rectified linear unit prelu, and the step size of the convolution layer is s=1) to extract features; then these features are input into the deep learning filter, and finally the luminance component filtered image Y' is obtained.

[0089] Optionally, in Fig. 9 In the illustrated embodiment, the deep learning filter includes a convolution unit, a residual unit, and a shuffle unit connected in sequence. The convolution unit includes a convolution layer with a step size of s=1, a parameterized rectified linear unit prelu, and a convolution layer with a step size of s=2; the residual unit includes N residual blocks connected in sequence; and the shuffle unit is a shuffle unit. Fig. 9 In the embodiment shown in , a convolution layer may be further arranged between the residual unit and the rearrangement unit shuffle, and the step length of the convolution layer is s=1.

[0090] exist Fig. 9In the illustrated embodiment, the chroma component information includes: the chroma component image before filtering (i.e., the chroma component reconstructed image), the chroma component predicted image, and the chroma component block division information. In other embodiments of the present application, the chroma component information may also include a portion of the chroma component image before filtering (i.e., the chroma component reconstructed image), the chroma component predicted image, and the chroma component block division information. If the chroma component information does not include the chroma component predicted image, then Fig. 9 In the embodiment shown, the feature extraction part of the predicted image (Y, Cb, Cr) by the convolution unit can be removed; if the chrominance component information does not include the chrominance component block division information, then Fig. 9 In the illustrated embodiment, the portion for extracting features from the block division information (Y, Cb, Cr) through the convolution unit may be removed.

[0091] In one embodiment of the present application, the N residual blocks included in the residual unit may be any positive integer number of residual blocks, such as 1, 2, or 3.

[0092] like Fig.10 As shown, the residual block structure in one embodiment of the present application may include: a first convolutional layer (whose convolution kernel size may be 1×1), a parameterized rectified linear unit, and a second convolutional layer (whose convolution kernel size may be 3×3) connected in sequence; wherein, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the second convolutional layer is used as the output of the residual block.

[0093] like Fig.11 As shown, the residual block structure in one embodiment of the present application may include: a first convolutional layer (whose convolution kernel size may be 1×1), a parameterized rectified linear unit, a second convolutional layer (whose convolution kernel size may be 3×3) and a convolutional block attention module (Convolutional Block Attention Module, referred to as CBAM) connected in sequence; wherein, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the convolutional block attention module is used as the output of the residual block.

[0094] like Fig.12 As shown, the residual block structure in one embodiment of the present application may include: a first convolutional layer (whose convolution kernel size may be 1×1), a parameterized rectified linear unit, a third convolutional layer (whose convolution kernel size may be 1×1) and a second convolutional layer (whose convolution kernel size may be 3×3) connected in sequence; wherein, the convolution kernel of the first convolutional layer is the same size as the convolution kernel of the third convolutional layer, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the second convolutional layer is used as the output of the residual block.

[0095] like Fig.13 As shown, the residual block structure in one embodiment of the present application may include: a first convolutional layer (whose convolution kernel size may be 1×1), a parameterized rectified linear unit, a third convolutional layer (whose convolution kernel size may be 1×1), a second convolutional layer (whose convolution kernel size may be 3×3) and a convolutional block attention unit connected in sequence; wherein, the convolution kernel of the first convolutional layer is the same size as the convolution kernel of the third convolutional layer, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the convolutional block attention unit is used as the output of the residual block.

[0096] It should be noted that, in an embodiment of the present application, the residual unit may include one or more residual blocks, and the structure of each residual block may be any of the above. The number of convolutional layers and prelu layers included in the convolutional unit of the deep learning filter can be set according to actual needs, and the number of convolutional layers and prelu layers included in the residual block can also be set according to actual needs. At the same time, the number of channels of different convolutional layers may be the same or different.

[0097] In step S730, the generated input parameters are input into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter.

[0098] In one embodiment of the present application, the deep learning filter needs to use the same parameters as those used in the application as input during training. Specifically, in the training stage, it is necessary to obtain the sample reconstructed image of the luminance component and the corresponding chrominance component information (the chrominance component information is adjusted according to the usage scenario of the deep learning filter, that is, it matches the parameters used by the deep learning filter when it is used), and then generate the input parameters for training based on the sample reconstructed image of the luminance component and the chrominance component information, and then input the obtained input parameters into the deep learning filter, and adjust the parameters of the deep learning filter based on the loss value between the output of the deep learning filter and the expected filtering result image corresponding to the sample reconstructed image of the luminance component, and repeat this process until the deep learning filter meets the convergence conditions.

[0099] exist Figure 7 Based on the filtering method based on deep learning shown in FIG. 1 , a video encoding method is also proposed in the embodiment of the present application. The video encoding method can be executed by a device with a computing and processing function, such as a terminal device or a server. The specific process is as follows Fig.14 As shown, the following steps S1410 to S1440 are included:

[0100] In step S1410, a luminance component reconstructed image corresponding to the encoded image and chrominance component information corresponding to the encoded image are obtained.

[0101] The specific implementation details of this step can refer to the aforementioned step S710 and will not be repeated here.

[0102] In step S1420, input parameters of the deep learning filter are generated according to the luminance component reconstructed image and the chrominance component information.

[0103] The specific implementation details of this step can refer to the aforementioned step S720 and will not be repeated here.

[0104] In step S1430, the generated input parameters are input into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter.

[0105] In step S1440, a brightness component prediction image corresponding to the next frame of image is generated based on the filtered image of the brightness component reconstructed image, and encoding processing is performed on the next frame of video image based on the brightness component prediction image.

[0106] Optionally, after generating the filtered image, you can refer to Figure 3 The process shown in FIG. 1 is to use the filtered image of the brightness component reconstructed image as the reference image of the brightness component of the next frame image for motion estimation and motion compensation prediction, and then obtain the brightness component prediction image of the next frame image based on the result of the motion compensation prediction and the intra-frame prediction result, and continue to repeat the process. Figure 3 The process shown in is continued until the encoding of the video image is completed.

[0107] Accordingly, in Figure 7 Based on the filtering method based on deep learning shown in FIG. 1 , a video decoding method is also proposed in the embodiment of the present application. The video decoding method can be executed by a device with a computing and processing function, such as a terminal device or a server. The specific process is as follows Fig.15 As shown, the following steps S1510 to S1540 are included:

[0108] In step S1510, a luminance component reconstructed image corresponding to the encoded image and chrominance component information corresponding to the encoded image are obtained.

[0109] The specific implementation details of this step can refer to the aforementioned step S710 and will not be repeated here.

[0110] In step S1520, input parameters of the deep learning filter are generated according to the luminance component reconstructed image and the chrominance component information.

[0111] The specific implementation details of this step can refer to the aforementioned step S720 and will not be repeated here.

[0112] In step S1530, the generated input parameters are input into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter.

[0113] In step S1540, a luminance component prediction image corresponding to the next frame image is generated based on the filtered image of the luminance component reconstructed image, and the video code stream is decoded based on the luminance component prediction image.

[0114] Optionally, after generating a filtered image of a luminance component reconstructed image, the filtered image of the luminance component reconstructed image can be used as a luminance component reference image of the next frame for motion estimation and motion compensation prediction, and then based on the result of the motion compensation prediction and the result of the intra-frame prediction, the luminance component predicted image of the next frame is obtained, and the luminance component predicted image and the luminance component reconstructed residual signal obtained by inverse quantization and inverse transformation are superimposed again to generate the luminance component reconstructed image of the next frame, and this process is repeated to realize the decoding processing of the video code stream.

[0115] The technical solution of the embodiment of the present application makes it possible to fully utilize the information of the chrominance component when filtering the luminance component of an image, and further utilize the existing chrominance component information to improve the performance of the deep learning filter for the luminance component, thereby improving the filtering effect and being beneficial to improving the encoding and decoding efficiency of the video.

[0116] The following describes an apparatus embodiment of the present application, which can be used to execute the deep learning-based filtering method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the deep learning-based filtering method in the present application.

[0117] Fig.16 A block diagram of a deep learning-based filtering device according to an embodiment of the present application is shown. The deep learning-based filtering device can be set in a device with a computing and processing function, such as a terminal device or a server.

[0118] Reference Fig.16 As shown, a deep learning-based filtering device 1600 according to an embodiment of the present application includes: an acquisition unit 1602 , a generation unit 1604 and a processing unit 1606 .

[0119] Among them, the acquisition unit 1602 is configured to obtain a luminance component reconstructed image corresponding to the encoded image and chrominance component information corresponding to the encoded image; the generation unit 1604 is configured to generate input parameters of the deep learning filter according to the luminance component reconstructed image and the chrominance component information; the processing unit 1606 is configured to input the input parameters to the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter.

[0120] In some embodiments of the present application, based on the aforementioned scheme, the chrominance component information includes at least one of the following: a chrominance component pre-filtering image corresponding to the encoded image, a chrominance component prediction image corresponding to the encoded image, and chrominance component block division information corresponding to the encoded image.

[0121] In some embodiments of the present application, based on the aforementioned scheme, the generation unit 1604 is configured to: upsample the chrominance component information to obtain chrominance component information of the same size as the luminance component reconstructed image; perform layer merging processing on the luminance component reconstructed image and the upsampled chrominance component information, and use the result of the layer merging processing as the input parameter of the deep learning filter.

[0122] In some embodiments of the present application, based on the aforementioned scheme, the generation unit 1604 is configured to: upsample the chrominance component information to obtain chrominance component information of the same size as the luminance component reconstructed image, the chrominance component information including: a chrominance component pre-filtering image, a chrominance component prediction image, and chrominance component block division information; merge the luminance component reconstructed image with the chrominance component pre-filtering image after upsampling, and extract the merged image features to obtain a first feature; merge the luminance component prediction image corresponding to the encoded image with the chrominance component prediction image after upsampling, and extract the merged image features to obtain a second feature; merge the luminance component block division information corresponding to the encoded image with the chrominance component block division information after upsampling, and extract the merged image features to obtain a third feature; generate the input parameters according to the first feature, the second feature, and the third feature.

[0123] In some embodiments of the present application, based on the aforementioned scheme, the deep learning filter includes a convolution unit, a residual unit and a rearrangement unit connected in sequence, and the residual unit includes at least one residual block.

[0124] In some embodiments of the present application, based on the aforementioned scheme, if the residual unit includes multiple residual blocks, the number of channels of the multiple residual blocks is the same, or the number of channels of each residual block in the multiple residual blocks is not completely the same.

[0125] In some embodiments of the present application, based on the aforementioned scheme, a residual block includes: a first convolutional layer, a parameterized rectified linear unit, and a second convolutional layer connected in sequence; wherein the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the second convolutional layer is used as the output of the residual block.

[0126] In some embodiments of the present application, based on the aforementioned scheme, a residual block includes: a first convolutional layer, a parameterized rectified linear unit, a second convolutional layer and a convolutional block attention unit connected in sequence; wherein, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the convolutional block attention unit is used as the output of the residual block.

[0127] In some embodiments of the present application, based on the aforementioned scheme, a residual block includes: a first convolutional layer, a parameterized rectified linear unit, a third convolutional layer, and a second convolutional layer connected in sequence; wherein the convolution kernel of the first convolutional layer is the same size as the convolution kernel of the third convolutional layer, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the second convolutional layer is used as the output of the residual block.

[0128] In some embodiments of the present application, based on the aforementioned scheme, a residual block includes: a first convolutional layer, a parameterized rectified linear unit, a third convolutional layer, a second convolutional layer and a convolutional block attention unit connected in sequence; wherein the convolution kernel of the first convolutional layer is the same size as the convolution kernel of the third convolutional layer, the input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the convolutional block attention unit is used as the output of the residual block.

[0129] Fig.17 A block diagram of a video encoding device according to an embodiment of the present application is shown. The video encoding device can be set in a device with a computing and processing function, such as a terminal device or a server.

[0130] Reference Fig.17 As shown, a video encoding device 1700 according to an embodiment of the present application includes: an acquisition unit 1602 , a generation unit 1604 , a processing unit 1606 and an encoding unit 1702 .

[0131] Among them, the acquisition unit 1602 is configured to obtain a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; the generation unit 1604 is configured to generate input parameters of a deep learning filter according to the luminance component reconstructed image and the chrominance component information; the processing unit 1606 is configured to input the input parameters into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter; the encoding unit 1702 is configured to generate a luminance component predicted image corresponding to a next frame of image based on the filtered image, and encode the next frame of video image based on the luminance component predicted image.

[0132] Fig.18 A block diagram of a video decoding device according to an embodiment of the present application is shown. The video decoding device can be set in a device with a computing and processing function, such as a terminal device or a server.

[0133] Reference Fig.18 As shown, a video decoding device 1800 according to an embodiment of the present application includes: an acquisition unit 1602 , a generation unit 1604 , a processing unit 1606 and a decoding unit 1802 .

[0134] Among them, the acquisition unit 1602 is configured to obtain a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; the generation unit 1604 is configured to generate input parameters of a deep learning filter according to the luminance component reconstructed image and the chrominance component information; the processing unit 1606 is configured to input the input parameters into the deep learning filter to obtain a filtered image corresponding to the luminance component reconstructed image output by the deep learning filter; the decoding unit 1802 is configured to generate a luminance component predicted image corresponding to the next frame image based on the filtered image, and decode the video code stream based on the luminance component predicted image.

[0135] Fig.19 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0136] It should be noted that Fig.19 The computer system 1900 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0137] like Fig.19As shown, the computer system 1900 includes a central processing unit (CPU) 1901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1902 or the program loaded from the storage part 1908 to the random access memory (RAM) 1903, such as executing the method described in the above embodiment. In RAM 1903, various programs and data required for system operation are also stored. CPU 1901, ROM 1902 and RAM 1903 are connected to each other through bus 1904. Input / output (I / O) interface 1905 is also connected to bus 1904.

[0138] The following components are connected to the I / O interface 1905: an input section 1906 including a keyboard, a mouse, etc.; an output section 1907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1908 including a hard disk, etc.; and a communication section 1909 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1909 performs communication processing via a network such as the Internet. A drive 1910 is also connected to the I / O interface 1905 as needed. A removable medium 1911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1910 as needed so that a computer program read therefrom is installed into the storage section 1908 as needed.

[0139] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1909, and / or installed from a removable medium 1911. When the computer program is executed by a central processing unit (CPU) 1901, various functions defined in the system of the present application are executed.

[0140] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0141] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0143] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0144] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0145] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0146] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0147] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A filtering method based on deep learning, It is characterized in that include: Obtaining a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; Upsampling the chrominance component information to obtain chrominance component information of the same size as the luminance component reconstructed image; The chroma component information includes: a chroma component pre-filtered image, a chroma component predicted image, and chroma component block division information; Performing layer merging processing on the reconstructed image of the luminance component and the pre-filtered image of the chrominance component after upsampling processing, and extracting the image features after the layer merging to obtain a first feature; Performing layer merging processing on the luminance component prediction image and the chrominance component prediction image after upsampling processing corresponding to the encoded image, and extracting image features after the layer merging to obtain a second feature; Performing layer merging processing on the luminance component block division information corresponding to the encoded image and the chrominance component block division information after upsampling processing, and extracting image features after the layer merging to obtain a third feature; Generate an input parameter of a deep learning filter according to the first feature, the second feature and the third feature; The input parameters are input into the deep learning filter to obtain a filtered image corresponding to the brightness component reconstructed image output by the deep learning filter.

2. The filtering method based on deep learning according to claim 1, It is characterized in that The deep learning filter includes a convolution unit, a residual unit and a rearrangement unit connected in sequence, and the residual unit includes at least one residual block.

3. The filtering method based on deep learning according to claim 2, It is characterized in that If the residual unit includes a plurality of residual blocks, the plurality of residual blocks have the same number of channels, or the number of channels of each residual block in the plurality of residual blocks is not completely the same.

4. The filtering method based on deep learning according to claim 2, It is characterized in that One of the residual blocks comprises: a first convolutional layer, a parameterized rectified linear unit and a second convolutional layer connected in sequence; The input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the second convolutional layer is used as the output of the residual block.

5. The filtering method based on deep learning according to claim 2, It is characterized in that One of the residual blocks includes: a first convolutional layer, a parameterized rectified linear unit, a second convolutional layer and a convolutional block attention unit connected in sequence; The input of the first convolutional layer is used as the input of the residual block, and the superposition result of the input of the first convolutional layer and the output of the attention unit of the convolutional block is used as the output of the residual block.

6. The filtering method based on deep learning according to claim 2, It is characterized in that One of the residual blocks comprises: a first convolutional layer, a parameterized rectified linear unit, a third convolutional layer and a second convolutional layer connected in sequence; Among them, the convolution kernel of the first convolution layer has the same size as the convolution kernel of the third convolution layer, the input of the first convolution layer is used as the input of the residual block, and the superposition result of the input of the first convolution layer and the output of the second convolution layer is used as the output of the residual block.

7. The filtering method based on deep learning according to claim 2, It is characterized in that One of the residual blocks includes: a first convolutional layer, a parameterized rectified linear unit, a third convolutional layer, a second convolutional layer and a convolutional block attention unit connected in sequence; Among them, the convolution kernel of the first convolution layer has the same size as the convolution kernel of the third convolution layer, the input of the first convolution layer is used as the input of the residual block, and the superposition result of the input of the first convolution layer and the output of the attention unit of the convolution block is used as the output of the residual block.

8. A video encoding method, It is characterized in that include: Obtaining a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; Upsampling the chrominance component information to obtain chrominance component information of the same size as the luminance component reconstructed image; The chroma component information includes: a chroma component pre-filtered image, a chroma component predicted image, and chroma component block division information; Performing layer merging processing on the reconstructed image of the luminance component and the pre-filtered image of the chrominance component after upsampling processing, and extracting the image features after the layer merging to obtain a first feature; Performing layer merging processing on the luminance component prediction image and the chrominance component prediction image after upsampling processing corresponding to the encoded image, and extracting image features after the layer merging to obtain a second feature; Performing layer merging processing on the luminance component block division information corresponding to the encoded image and the chrominance component block division information after upsampling processing, and extracting image features after the layer merging to obtain a third feature; Generate an input parameter of a deep learning filter according to the first feature, the second feature and the third feature; Inputting the input parameters into the deep learning filter to obtain a filtered image corresponding to the brightness component reconstructed image output by the deep learning filter; A brightness component prediction image corresponding to a next frame of image is generated based on the filtered image, and encoding processing is performed on the next frame of video image based on the brightness component prediction image.

9. A video decoding method, It is characterized in that include: Obtaining a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; Upsampling the chrominance component information to obtain chrominance component information of the same size as the luminance component reconstructed image; The chroma component information includes: a chroma component pre-filtered image, a chroma component predicted image, and chroma component block division information; Performing layer merging processing on the reconstructed image of the luminance component and the pre-filtered image of the chrominance component after upsampling processing, and extracting the image features after the layer merging to obtain a first feature; Performing layer merging processing on the luminance component prediction image and the chrominance component prediction image after upsampling processing corresponding to the encoded image, and extracting image features after the layer merging to obtain a second feature; Performing layer merging processing on the luminance component block division information corresponding to the encoded image and the chrominance component block division information after upsampling processing, and extracting image features after the layer merging to obtain a third feature; Generate an input parameter of a deep learning filter according to the first feature, the second feature and the third feature; Inputting the input parameters into the deep learning filter to obtain a filtered image corresponding to the brightness component reconstructed image output by the deep learning filter; A brightness component prediction image corresponding to a next frame of image is generated based on the filtered image, and a video code stream is decoded based on the brightness component prediction image.

10. A filtering device based on deep learning, It is characterized in that include: An acquisition unit configured to acquire a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; A generating unit configured to perform upsampling processing on the chrominance component information to obtain chrominance component information of the same size as the luminance component reconstructed image; The chroma component information includes: a chroma component pre-filtering image, a chroma component prediction image, and chroma component block division information; performing layer merging processing on the luminance component reconstructed image and the chroma component pre-filtering image after upsampling processing, and extracting the image features after the layer merging to obtain a first feature; performing layer merging processing on the luminance component prediction image corresponding to the encoded image and the chroma component prediction image after upsampling processing, and extracting the image features after the layer merging to obtain a second feature; performing layer merging processing on the luminance component block division information corresponding to the encoded image and the chroma component block division information after upsampling processing, and extracting the image features after the layer merging to obtain a third feature; generating input parameters of a deep learning filter according to the first feature, the second feature, and the third feature; The processing unit is configured to input the input parameters into the deep learning filter to obtain a filtered image output by the deep learning filter corresponding to the brightness component reconstructed image.

11. A video encoding device, It is characterized in that include: An acquisition unit configured to acquire a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; A generating unit configured to perform upsampling processing on the chrominance component information to obtain chrominance component information of the same size as the luminance component reconstructed image; The chroma component information includes: a chroma component pre-filtering image, a chroma component prediction image, and chroma component block division information; performing layer merging processing on the luminance component reconstructed image and the chroma component pre-filtering image after upsampling processing, and extracting the image features after the layer merging to obtain a first feature; performing layer merging processing on the luminance component prediction image corresponding to the encoded image and the chroma component prediction image after upsampling processing, and extracting the image features after the layer merging to obtain a second feature; performing layer merging processing on the luminance component block division information corresponding to the encoded image and the chroma component block division information after upsampling processing, and extracting the image features after the layer merging to obtain a third feature; generating input parameters of a deep learning filter according to the first feature, the second feature, and the third feature; a processing unit configured to input the input parameters into the deep learning filter to obtain a filtered image output by the deep learning filter corresponding to the brightness component reconstructed image; The encoding unit is configured to generate a brightness component prediction image corresponding to a next frame of image based on the filtered image, and to perform encoding processing on the next frame of video image based on the brightness component prediction image.

12. A video decoding device, It is characterized in that include: An acquisition unit configured to acquire a luminance component reconstructed image corresponding to an encoded image and chrominance component information corresponding to the encoded image; A generating unit configured to perform upsampling processing on the chrominance component information to obtain chrominance component information of the same size as the luminance component reconstructed image; The chroma component information includes: a chroma component pre-filtering image, a chroma component prediction image, and chroma component block division information; performing layer merging processing on the luminance component reconstructed image and the chroma component pre-filtering image after upsampling processing, and extracting the image features after the layer merging to obtain a first feature; performing layer merging processing on the luminance component prediction image corresponding to the encoded image and the chroma component prediction image after upsampling processing, and extracting the image features after the layer merging to obtain a second feature; performing layer merging processing on the luminance component block division information corresponding to the encoded image and the chroma component block division information after upsampling processing, and extracting the image features after the layer merging to obtain a third feature; generating input parameters of a deep learning filter according to the first feature, the second feature, and the third feature; a processing unit configured to input the input parameters into the deep learning filter to obtain a filtered image output by the deep learning filter corresponding to the brightness component reconstructed image; The decoding unit is configured to generate a brightness component prediction image corresponding to a next frame of image based on the filtered image, and decode the video code stream based on the brightness component prediction image.

13. A computer readable medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, it implements the deep learning-based filtering method as described in any one of claims 1 to 7, or implements the video encoding method as described in claim 8, or implements the video decoding method as described in claim 9.

14. An electronic device, It is characterized in that include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the deep learning-based filtering method as described in any one of claims 1 to 7, or implement the video encoding method as described in claim 8, or implement the video decoding method as described in claim 9.

15. A computer program product, It is characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium. The processor of the electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the deep learning-based filtering method described in any one of claims 1 to 7, or implements the video encoding method described in claim 8, or implements the video decoding method described in claim 9.

Images