Loop filtering implementation method and device and computer storage medium

Abstract

The embodiment of the invention discloses a loop filtering implementation method and device and a computer storage medium. The method comprises the steps of obtaining a to-be-filtered image; wherein the to-be-filtered image is generated by an original image in a to-be-coded video in a video coding process, the to-be-coded video comprises an original image frame, and the original image frame comprises the original image; respectively obtaining at least two image components of the to-be-filtered image based on the to-be-filtered image; determining fusion information of the to-be-filtered image; wherein the fusion information at least comprises fusion of the at least two image components; and performing loop filtering processing on the to-be-filtered image based on the fusion information to obtain at least one image component of the to-be-filtered image after filtering.

Description

[0001] Divisional Application Explanation

[0002] This application claims the divisional application of PCT international patent application PCT / CN2019 / 077372 filed on March 7, 2019, the Chinese patent application number 201980090702.4 upon entering the Chinese national phase, and the invention title "Method, Apparatus and Computer Storage Medium for Implementing Loop Filtering". Technical Field

[0003] The embodiments of this application relate to the field of image processing technology, and particularly to a method, apparatus and computer storage medium for implementing loop filtering. Background Art

[0004] In a video codec system, most video encodings adopt a hybrid coding framework based on block-shaped coding units (CUs). Since adjacent CUs adopt different coding parameters, such as different transformation processes, different quantization parameters (QPs), different prediction methods, different reference image frames, etc., and the errors introduced by each CU are independent in terms of their magnitudes and distribution characteristics, discontinuities at the boundaries of adjacent CUs cause blocking artifacts, which affect the subjective and objective quality of the reconstructed image and even the prediction accuracy of subsequent encoding and decoding.

[0005] Thus, during the encoding and decoding process, a loop filter is used to improve the subjective and objective quality of the reconstructed image. Traditional loop filters usually artificially summarize the characteristics of distorted images and artificially design the filter structure and configure the filter coefficients, such as deblocking filtering, sample adaptive offset, and adaptive loop filtering. These filters that rely on artificial design do not fit the optimal filter well, have poor adaptability and filtering effects, and the encoder needs to write filter-related parameters that depend on local statistical information into the bitstream to ensure consistency between the encoder and decoder, which increases the coding bit count.

[0006] With the rapid development of deep learning theory, convolutional neural networks (CNNs) have been proposed in the industry to filter the reconstructed image to remove image distortion, achieving a significant improvement in subjective and objective quality compared to traditional loop filters. However, the current CNN filters do not fully utilize relevant information, resulting in limited improvement in the subjective and objective quality of the reconstructed image. Summary of the Invention

[0007] An embodiment of the present application provides a method, apparatus, and computer storage medium for implementing loop filtering. By adopting a cascaded processing structure and fusing coding parameters such as block partitioning information and / or QP information as auxiliary information with multiple input image components, not only is the relationship between multiple image components fully utilized, but also the computational complexity is reduced, and the coding bit rate is saved. At the same time, the subjective and objective quality of the video reconstructed image during the encoding and decoding process is further improved.

[0008] The technical solution of the embodiment of the present application can be implemented as follows:

[0009] In a first aspect, an embodiment of the present application provides a method for implementing loop filtering, the method comprising:

[0010] Obtain an image to be filtered; wherein, the image to be filtered is generated during the video encoding process of the original image in the video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image;

[0011] Based on the image to be filtered, obtain at least two image components of the image to be filtered respectively;

[0012] Determine the fusion information of the image to be filtered; wherein, the fusion information at least includes the information obtained by fusing the at least two image components;

[0013] Based on the fusion information, perform loop filtering processing on the image to be filtered to obtain at least one image component of the filtered image to be filtered.

[0014] In a second aspect, an embodiment of the present application provides a device for implementing loop filtering, the device for implementing loop filtering comprising: an obtaining unit, a splitting unit, a determining unit, and a filtering unit, wherein,

[0015] The obtaining unit is configured to obtain an image to be filtered; wherein, the image to be filtered is generated during the video encoding process of the original image in the video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image;

[0016] The splitting unit is configured to, based on the image to be filtered, obtain at least two image components of the image to be filtered respectively;

[0017] The determining unit is configured to determine the fusion information of the image to be filtered; wherein, the fusion information at least includes the information obtained by fusing the at least two image components;

[0018] The filtering unit is configured to, based on the fusion information, perform loop filtering processing on the image to be filtered to obtain at least one image component of the filtered image to be filtered.

[0019] In a third aspect, an embodiment of the present application provides a loop filtering implementation device, where the loop filtering implementation device includes: a memory and a processor, where,

[0020] The memory is used to store a computer program that can run on the processor;

[0021] The processor is configured to execute the steps of the method described in the first aspect when running the computer program.

[0022] In a fourth aspect, an embodiment of the present application provides a computer storage medium, where the computer storage medium stores a loop filtering implementation program, and when the loop filtering implementation program is executed by at least one processor, the steps of the method described in the first aspect are implemented.

[0023] An embodiment of the present application provides a loop filtering implementation method, device, and computer storage medium. First, a to-be-filtered image is obtained, and the to-be-filtered image is an original image in a to-be-encoded video generated during the video encoding process; then, based on the to-be-filtered image, at least two image components of the to-be-filtered image are respectively obtained; this processing process can be regarded as a splitting stage; then, the fusion information of the to-be-filtered image is determined; where the fusion information at least includes that obtained by fusing the at least two image components; this processing process can be regarded as a merging stage; in this way, by adopting a cascaded processing structure and performing fusion processing on multiple input image components, not only the relationship between the multiple image components is fully utilized, but also the problem of multiple complete network forward calculations required for these multiple image components is effectively avoided, thereby reducing the computational complexity and saving the coding bit rate; finally, loop filtering processing is performed on the to-be-filtered image based on the fusion information to obtain at least one image component of the filtered to-be-filtered image; in this way, the fusion information can further assist in filtering, improving the subjective and objective quality of the video reconstructed image during the encoding and decoding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the composition structure of a traditional coding block diagram provided for related technical solutions;

[0025] Figure 2 A schematic diagram of the composition structure of an improved coding block diagram provided for an embodiment of the present application;

[0026] Figure 3 A schematic flowchart of a loop filtering implementation method provided for an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of a block partitioning matrix provided for an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the composition structure of a traditional CNN filter provided by an embodiment of the present application;

[0029] Figure 6A and Figure 6B Another schematic diagram of the composition structure of a traditional CNN filter provided by an embodiment of the present application;

[0030] Figure 7 A schematic diagram of the composition structure of a loop filter framework provided by an embodiment of the present application;

[0031] Figure 8 Another schematic diagram of the composition structure of a loop filter framework provided by an embodiment of the present application;

[0032] Figure 9 A schematic diagram of the composition structure of a loop filter implementation device provided by an embodiment of the present application;

[0033] Figure 10 A schematic diagram of the specific hardware structure of a loop filter implementation device provided by an embodiment of the present application. Detailed implementation manners

[0034] In order to understand the features and technical content of the embodiments of the present application in more detail, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are only for reference and illustration purposes and are not used to limit the embodiments of the present application.

[0035] In a video coding and decoding system, the video to be encoded includes original image frames, and the original image frames include original images, which are subjected to various processes such as prediction, transformation, quantization, reconstruction, and filtering. During these processes, the processed video images may have pixel value offsets relative to the original images, resulting in visual impairments or artifacts. In addition, under the hybrid coding framework based on block-shaped CUs adopted by most video coding and decoding systems, since adjacent coding blocks use different coding parameters (such as different transformation processes, different QPs, different prediction methods, different reference image frames, etc.), the error sizes and their distribution characteristics introduced by each coding block are independent of each other, and the discontinuity at the boundaries of adjacent coding blocks generates blocking artifacts. These distortions not only affect the subjective and objective quality of the reconstructed images, but if the reconstructed images are used as reference images for subsequent coding pixels, they will even affect the prediction accuracy of subsequent coding and decoding, and thus affect the size of the bits in the video bitstream. Therefore, in a video coding and decoding system, an in-loop filter is often added to improve the subjective and objective quality of the reconstructed images.

[0036] See Figure 1 , which shows the composition structure diagram of a traditional coding frame provided by a related technical solution. Figure 10 AsFigure 1 As shown, this traditional coding frame Figure 10 may include components such as a transform and quantization unit 101, an inverse transform and inverse quantization unit 102, a prediction unit 103, a filtering unit 104, and an entropy coding unit 105. Among them, the prediction unit 103 further includes an intra prediction unit 1031 and an inter prediction unit 1032. For the input original image, coding tree units (CTUs) can be obtained through preliminary partitioning. Continuing with content-adaptive partitioning of a CTU can obtain CUs, and a CU generally contains one or more coding blocks (CBs). Performing intra prediction by the intra prediction unit 1031 or inter prediction by the inter prediction unit 1032 on the coding block can obtain residual information. The transform and quantization unit 101 is used to perform a transform on the coding block for the residual information, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate. After determining the prediction mode, the prediction unit 103 is further used to provide the selected intra prediction data or inter prediction data to the entropy coding unit 105. In addition, the inverse transform and inverse quantization unit 102 is used for the reconstruction of the coding block, reconstructing the residual block in the pixel domain. The reconstructed residual block removes block effect artifacts through the filtering unit 104, and then the reconstructed residual block is added to the decoded image buffer unit to generate a reconstructed reference image. The entropy coding unit 105 is used to encode various coding parameters and the quantized transform coefficients. For example, the entropy coding unit 105 uses header information coding and context-based adaptive binary arithmetic coding (CABAC) algorithm, and can be used to encode the coding information indicating the determined prediction mode and output the corresponding bitstream.

[0037] Regarding Figure 1 the above-mentioned traditional coding frame Figure 10, the filtering unit 104 is a loop filter, also known as an in-loop filter, which may include a de-blocking filter (DBF) 1041, a sample adaptive offset (SAO) filter 1042, an adaptive loop filter (ALF) 1043, etc. Among them, the de-blocking filter 1041 is used to implement de-blocking filtering. In the next-generation video coding standard H.266 / versatile video coding (VVC), for all coding block boundaries in the original image, first, the boundary strength is determined based on the coding parameters on both sides of the boundary, and whether to perform de-blocking filtering decision is judged according to the calculated block boundary texture degree value. Finally, the pixel information on both sides of the coding block boundary is corrected according to the boundary strength and filtering decision. In VVC, after the de-blocking filtering is executed, in order to reduce the quantization distortion of high-frequency AC coefficients, the SAO technology is also introduced, that is, the sample adaptive offset filter 1042; further, starting from the pixel domain, negative values are added to the pixels at the peaks and positive values are added to the pixels at the valleys for compensation processing. In VVC, after the de-blocking filtering and the sample adaptive offset filtering are executed, the adaptive loop filter 1043 needs to be further used for filtering processing; for the adaptive loop filtering, it calculates the optimal filter in the mean square sense based on the pixel values of the original image and the pixel values of the distorted image. However, these filters (such as the de-blocking filter 1041, the sample adaptive offset filter 1042, and the adaptive loop filter 1043, etc.) not only require delicate manual design and a large number of judgment decisions; but also at the encoding end, the filter-related parameters depending on local statistical information (such as filter coefficients and the flag value indicating whether to select this filter, etc.) need to be written into the bitstream to ensure the consistency between the encoding end and the decoding end, increasing the encoding bit count; at the same time, the filters designed manually do not fit well with the complex function of the real optimization objective, and the filtering effect needs to be enhanced.

[0038] The embodiment of the present application provides a method for implementing loop filtering, which is applied to an improved coding block diagram; compared with Figure 1 the traditional coding block Figure 10In comparison, the main difference is that an improved loop filter is used to replace the deblocking filter 1041, sample adaptive compensation filter 1042, adaptive loop filter 1043, etc. in the related technical solutions. In the embodiments of the present application, the improved loop filter may be a convolutional neural network (CNN) filter, or may also be a filter established by other deep learning, and the embodiments of the present application do not make specific limitations.

[0039] Taking the convolutional neural network filter as an example, refer to Figure 2 , which shows a schematic structural diagram of the composition of an improved coding block diagram 20 provided by the embodiments of the present application. As Figure 2 shown, compared with the traditional coding block Figure 10 , the filtering unit 104 in the improved coding block diagram 20 includes a convolutional neural network filter 201. This convolutional neural network filter 201 can not only completely replace Figure 1 the deblocking filter 1041, sample adaptive compensation filter 1042, and adaptive loop filter 1043, but can also partially replace Figure 1 any one or two of the deblocking filter 1041, sample adaptive compensation filter 1042, and adaptive loop filter 1043 in Figure 1 , and can even be used in combination with any one or more of the deblocking filter 1041, sample adaptive compensation filter 1042, and adaptive loop filter 1043 in Figure 1 or Figure 2 . It should also be noted that for each component shown in Figure 1 or Figure 2 , such as the transform and quantization unit 101, inverse transform and inverse quantization unit 102, prediction unit 103, filtering unit 104, entropy coding unit 105, or convolutional neural network filter 201, these components may be virtual modules or hardware modules. In addition, those skilled in the art can understand that these units do not limit the coding block diagram, and the coding block diagram may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0040] In the embodiments of the present application, after the convolutional neural network filter 201 undergoes filtering network training, it can be directly deployed at the encoding end and the decoding end, so that there is no need to transmit any filter-related parameters; moreover, this convolutional neural network filter 201 can also fuse auxiliary information such as block partitioning information and / or QP information with multiple input image components; in this way, not only the relationship between multiple image components is fully utilized, but also the computational complexity is reduced, the coding rate is saved; at the same time, the subjective and objective quality of the video reconstructed image during the encoding and decoding process is further improved.

[0041] It should be noted that the loop filtering implementation method of the embodiments of the present application can be applied not only to the encoding system, but also to the decoding system. Generally speaking, in order to save the encoding bit rate and ensure that the decoding system can perform correct decoding processing, the loop filter of the embodiments of the present application must be deployed synchronously in the encoding system and the decoding system. The following will take the application in the encoding system as an example for detailed description.

[0042] See Figure 3 , which shows a schematic flowchart of a loop filtering implementation method provided by the embodiments of the present application. The method may include:

[0043] S301: Obtain the image to be filtered; wherein, the image to be filtered is generated during the video encoding process of the original image in the video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image;

[0044] S302: Based on the image to be filtered, obtain at least two image components of the image to be filtered respectively;

[0045] S303: Determine the fusion information of the image to be filtered; wherein, the fusion information at least includes that obtained by fusing the at least two image components;

[0046] S304: Perform loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component after filtering the image to be filtered.

[0047] It should be noted that since the original image can be divided into CTUs, or the CTUs can be divided into CUs; that is to say, the block division information in the embodiments of the present application can refer to CTU division information or CU division information; thus, the loop filtering implementation method of the embodiments of the present application can be applied not only to loop filtering at the CU level, but also to loop filtering at the CTU level, and the embodiments of the present application do not make specific limitations.

[0048] In an embodiment of the present application, after obtaining the image to be filtered, at least two image components of the image to be filtered are respectively obtained based on the image to be filtered; this processing process can be regarded as a splitting stage, which is used to separately obtain at least two image components; then the fusion information of the image to be filtered is determined; wherein, the fusion information at least includes that obtained by fusing the at least two image components; this processing process can be regarded as a merging stage, which is used to fuse at least two image components; in this way, the embodiment of the present application adopts a cascaded processing structure. By performing fusion processing on multiple input image components, not only the relationship between multiple image components is fully utilized, but also the problem of multiple complete network forward calculations required for these multiple image components is effectively avoided, thereby reducing the computational complexity and saving the coding rate; finally, loop filtering processing is performed on the image to be filtered based on the fusion information to obtain at least one image component after filtering the image to be filtered; in this way, the fusion information can further assist in filtering, improving the subjective and objective quality of the video reconstructed image in the encoding and decoding process.

[0049] In some embodiments, the image components include a first image component, a second image component, and a third image component; wherein, the first image component represents a luminance component, the second image component represents a first chrominance component, and the third image component represents a second chrominance component.

[0050] It should be noted that in video images, the first image component, the second image component, and the third image component are generally used to represent the original image or the image to be filtered. Among them, in the luminance-chrominance component representation method, these three image components are respectively a luminance component, a blue chrominance (color difference) component, and a red chrominance (color difference) component; specifically, the luminance component is usually represented by the symbol Y, the blue chrominance component is usually represented by the symbol Cb, and can also be represented by U; the red chrominance component is usually represented by the symbol Cr, and can also be represented by V. In the embodiment of the present application, the first image component can be the luminance component Y, the second image component can be the blue chrominance component U, and the third image component can be the red chrominance component V, but the embodiment of the present application does not make specific limitations. Among them, at least one image component represents one or more of the first image component, the second image component, and the third image component, and at least two image components can be the first image component, the second image component, and the third image component, or the first image component and the second image component, or the first image component and the third image component, or even the second image component and the third image component, and the embodiment of the present application does not make specific limitations.

[0051] In the next-generation video coding standard VVC, its corresponding test model is the VVC Test Model (VTM). When conducting tests on VTM, the currently standard test sequences use the YUV 4:2:0 format. Each frame of the video to be encoded in this format can be composed of three image components: one luminance component (denoted by Y) and two chrominance components (denoted by U and V). Assuming the height of the original image in the video to be encoded is H and the width is W, then the size information corresponding to the first image component is H×W, and the size information corresponding to the second image component or the third image component is both It should be noted that the embodiments of this application will be described by taking the YUV 4:2:0 format as an example, but the loop filter implementation method of the embodiments of this application is also applicable to other sampling formats.

[0052] Taking the YUV 4:2:0 format as an example, since the size information of the first image component is different from that of the second image component or the third image component, in order to input the first image component and / or the second image component and / or the third image component into the loop filter model at one time, it is necessary to perform sampling or recombination processing on these three image components so that the spatial domain size information of the three image components is the same.

[0053] In some embodiments, pixel rearrangement processing (which can also be called downsampling processing) can be performed on the high-resolution image component so that the spatial domain size information of the three image components is the same. Specifically, before obtaining at least two image components of the image to be filtered based on the image to be filtered, the method further includes:

[0054] For at least two original image components of the image to be filtered, select the high-resolution image component;

[0055] Perform pixel rearrangement processing on the high-resolution image component.

[0056] It should be noted that before performing other processing on the three image components (such as the first image component, the second image component, and the third image component) included in the original image, these three image components are the original image components. If the first image component is the luminance component, the second image component is the first chrominance component, and the third image component is the second chrominance component; then the high-resolution image component is the first image component, and at this time, pixel rearrangement processing needs to be performed on the first image component. Exemplarily, taking a 2×2-sized original image as an example, converting it into 4 channels, that is, arranging the 2×2×1 tensor into a 1×1×4 tensor; then when the size information of the first image component of the original image is H×W, it can be converted into form before loop filtering; since the size information of the second image component and the third image component is both This enables the spatial dimension information of the three image components to be the same; subsequently, after merging the first image component, the second image component, and the third image component after pixel rearrangement processing, etc., these three image components are transformed into the form of and input into the improved loop filter.

[0057] In some embodiments, upsampling processing may also be performed on the low-resolution image component to make the spatial dimension information of the three image components the same. Specifically, before respectively obtaining at least two image components of the image to be filtered based on the image to be filtered, the method further includes:

[0058] For at least two original image components of the image to be filtered, select the low-resolution image component;

[0059] Perform upsampling processing on the low-resolution image component.

[0060] It should be noted that in addition to performing pixel rearrangement processing (i.e., downward adjustment) on the size information of the high-resolution image component, in the embodiments of the present application, upsampling processing (i.e., upward adjustment) may also be performed on the low-resolution image component. In addition, for the low-resolution image component, not only upsampling processing can be performed, but also deconvolution processing can be performed, and even super-resolution processing can be performed, etc. The effects of these three processes are the same, and the embodiments of the present application do not make specific limitations.

[0061] It should also be noted that before performing other processing on the three image components (such as the first image component, the second image component, and the third image component) included in the original image, these three image components are original image components. If the first image component is a luminance component, the second image component is a first chrominance component, and the third image component is a second chrominance component; then the low-resolution image component is the second image component or the third image component, and at this time, upsampling processing needs to be performed on the second image component or the third image component. Exemplarily, when the size information of the second image component and the third image component of the original image is both at this time, before loop filtering, it can be converted into the form of H×W through upsampling processing; since the size information of the first image component is H×W, this can also make the spatial dimension information of the three image components the same, and the upsampled second image component and the upsampled third image component will be consistent with the resolution of the first image component.

[0062] In some embodiments, the obtaining of the image to be filtered includes:

[0063] Performing video encoding processing on the original image in the video to be encoded, and using the generated reconstructed image as the image to be filtered; or,

[0064] Perform video encoding processing on the original image in the encoded video to generate a reconstructed image; perform a preset filtering process on the reconstructed image, and use the image after the preset filtering as the image to be filtered.

[0065] It should be noted that, based on the improved encoding block diagram 20, during the process of performing video encoding on the original image in the video to be encoded, when performing video encoding processing on the original image, it is subjected to CU partitioning, prediction, transformation, quantization, etc. processing, and in order to obtain a reference image for performing video encoding on the subsequent image to be encoded, inverse transformation, inverse quantization, reconstruction, filtering, etc. processing can also be performed. In this way, the image to be filtered in the embodiments of the present application can be the reconstructed image generated after the reconstruction process during the video encoding process, or the image after the preset filtering obtained by performing other preset filtering methods (such as deblocking filtering method) on the reconstructed image, and the embodiments of the present application do not make specific limitations.

[0066] In some embodiments, before respectively obtaining at least two image components of the image to be filtered, the method further includes:

[0067] Based on at least two original image components of the image to be filtered, respectively determine first auxiliary information corresponding to each original image component; wherein, the first auxiliary information at least includes block partitioning information and / or quantization parameter information.

[0068] It can be understood that the first auxiliary information can be used to assist filtering and improve the filtering quality. In the embodiments of the present application, the first auxiliary information can not only be block partitioning information (such as CU partitioning information and / or CTU partitioning information), but also quantization parameter information, and even motion vector (MV) information, prediction direction information, etc.; these information can be used as the first auxiliary information alone, or can be combined arbitrarily as the first auxiliary information, such as using the block partitioning information alone as the first auxiliary information, or using the block partitioning information and the quantization parameter information together as the first auxiliary information, or using the block partitioning information and the MV information together as the first auxiliary information, etc., and the embodiments of the present application do not make specific limitations.

[0069] Optionally, in some embodiments, the step of respectively obtaining at least two image components of the image to be filtered based on the image to be filtered includes:

[0070] Based on at least two original image components of the image to be filtered, respectively perform component processing on each of the at least two original image components, and after the processing, obtain at least two image components of the image to be filtered.

[0071] Optionally, in some embodiments, obtaining at least two image components of the image to be filtered respectively includes:

[0072] Based on at least two original image components of the image to be filtered, add each original image component in the at least two original image components to the first auxiliary information corresponding to each original image component respectively, and after processing, obtain at least two image components of the image to be filtered.

[0073] It should be noted that "obtaining at least two image components of the image to be filtered respectively" can be regarded as the first splitting stage. In this way, for at least two original image components of the image to be filtered, component processing (such as deep learning) can be performed on them respectively, so as to obtain at least two image components; in addition, the first auxiliary information corresponding to each original image component can also be added to the corresponding image component, so as to obtain at least two image components; that is to say, for the first splitting stage, the first auxiliary information can be added or not, and the embodiments of the present application do not make specific limitations.

[0074] Optionally, in some embodiments, determining the fusion information of the image to be filtered includes:

[0075] Perform a fusion process on at least two image components of the image to be filtered to obtain the fusion information of the image to be filtered.

[0076] Optionally, in some embodiments, determining the fusion information of the image to be filtered includes:

[0077] Perform a fusion process on at least two image components of the image to be filtered and the first auxiliary information corresponding to each original image component to obtain the fusion information of the image to be filtered.

[0078] It should be noted that "determining the fusion information of the image to be filtered" can be regarded as the merging stage. In this way, the fusion information of the image to be filtered can be directly obtained by fusing at least two image components, or can be obtained by jointly fusing at least two image components and the corresponding first auxiliary information; that is to say, for the merging stage, the first auxiliary information can be added or not, and the embodiments of the present application do not make specific limitations.

[0079] In related technical solutions, such as Figure 1The filters in the filtering unit 104 shown process the three image components independently and require three complete network forward calculations in total. However, in the loop filtering implementation method of the present application embodiment, through information fusion, these three image components can be input at one time, or any two of these three image components can be input simultaneously. The present application embodiment does not make specific limitations. In this way, the problem of multiple complete network forward calculations required for multiple image components is effectively avoided, and the computational complexity is reduced.

[0080] It can also be understood that since the original image can be divided into CTUs, or a CTU can be divided into CUs. Therefore, the loop filtering implementation method of the present application embodiment can be applied not only to loop filtering at the CU level (in this case, the block division information is CU division information), but also to loop filtering at the CTU level (in this case, the block division information is CTU division information). The present application embodiment does not make specific limitations. Hereinafter, the CU division information will be used as an example of the block division information for description.

[0081] In some embodiments, the separately determining the first auxiliary information corresponding to each image component includes:

[0082] Based on the original image in the video to be encoded, perform CU division on each of at least two image components of the original image to obtain CU division information, and use the CU division information as the block division information corresponding to each image component of the image to be filtered.

[0083] Further, in some embodiments, the using the CU division information as the block division information corresponding to each image component of the image to be filtered includes:

[0084] For the CU division information, fill a first value at each pixel position corresponding to the CU boundary and fill a second value at other pixel positions to obtain a first matrix corresponding to the CU division information; wherein, the first value is different from the second value;

[0085] Use the first matrix as the block division information corresponding to each image component of the image to be filtered.

[0086] It should be noted that the first value can be a preset numerical value, letter, etc., and the second value can also be a preset numerical value, letter, etc., and the first value is different from the second value; for example, the first value can be set to 2 and the second value can be set to 1, but the present application embodiment does not make specific limitations.

[0087] In the embodiments of the present application, the CU partition information can be used as the first auxiliary information to assist in filtering the image to be filtered. That is to say, in the process of video encoding the original image in the video to be encoded, the CU partition information can be fully utilized, and it is fused with at least two image components of the image to be filtered to guide the filtering.

[0088] Specifically, the CU partition information is converted into a Coding Unit Map (CUmap) and represented as a two-dimensional matrix, that is, the CUmap matrix, which is also the first matrix in the embodiments of the present application. That is to say, taking the first image component of the original image as an example, it can be divided into multiple CUs. Each pixel position corresponding to the CU boundary is filled with a first value, and other pixel positions are filled with a second value, so that a first matrix reflecting the CU partition information can be constructed. Exemplarily, referring to Figure 4 , which shows a schematic structural diagram of a block partition matrix provided by the embodiments of the present application. As Figure 4 shown, if this figure represents a CTU, then this CTU can be divided into 9 CUs; assuming that the first value is set to 2 and the second value is set to 1; in this way, each pixel position corresponding to the CU boundary is filled with 2, and other pixel positions are filled with 1. That is to say, the pixel positions filled with 2 represent the CU boundaries, so that the CU partition information can be determined, that is, the first auxiliary information corresponding to the first image component of the image to be filtered.

[0089] It should also be noted that if the first image component is a luminance component and the second and third image components are both chrominance components, then the CU partition information of the first image component may be different from the CU partition information of the second image component or the third image component. Therefore, when the CU partition information of the first image component is different from the CU partition information of the second image component or the third image component, it is necessary to separately determine the CU partition information corresponding to the first image component of the image to be filtered and the CU partition information corresponding to the second image component or the third image component of the image to be filtered; then add it as the first auxiliary information to the corresponding first image component or second image component or third image component; when the CU partition information of the first image component is the same as the CU partition information of the second image component or the third image component, at this time, only the CU partition information of the first image component or the second image component or the third image component needs to be determined, and then the determined CU partition information is added as the first auxiliary information to the corresponding first image component or second image component or third image component; in this way, it is convenient to fuse the at least two new image components obtained later to perform loop filtering processing on the image to be filtered.

[0090] In some embodiments, the separately determining the first auxiliary information corresponding to each image component includes:

[0091] Based on the original image in the video to be encoded, the quantization parameter corresponding to each image component in at least two image components of the original image is separately obtained, and the quantization parameter is used as the quantization parameter information corresponding to each image component of the image to be filtered.

[0092] Further, in some embodiments, the using the quantization parameter as the quantization parameter information corresponding to each image component of the image to be filtered includes:

[0093] Second matrices having the same size as each image component of the original image are separately established; wherein, the normalized value of the quantization parameter corresponding to each image component of the original image is filled at each pixel position in the second matrix;

[0094] The second matrix is used as the quantization parameter information corresponding to each image component of the image to be filtered.

[0095] It should be noted that for images to be filtered corresponding to different quantization parameters, their distortion degrees are not the same. If the quantization parameter information is incorporated, then the filtering network can be enabled to adaptively have the ability to process any quantization parameter during the training process.

[0096] In the embodiments of the present application, the quantization parameter information can also be used as the first auxiliary information to assist in filtering the image to be filtered. That is to say, during the video encoding of the original image in the video to be encoded, the quantization parameter information can be fully utilized and fused with at least two image components of the image to be filtered to guide the filtering. Among them, the quantization parameter information can be normalized, and the quantization parameter information can also be non-normalized (such as classification processing, interval partitioning processing, etc.); the following will take the normalization processing of the quantization parameter as an example for detailed description.

[0097] Specifically, the quantization parameter information is converted into a second matrix reflecting the quantization parameter information; that is to say, taking the first image component of the original image as an example, a matrix having the same size as the first image component of the original image is established, and the normalized value of the quantization parameter corresponding to the first image component of the original image is filled at each pixel position in the matrix; wherein, the normalized value of the quantization parameter is represented by QP max (x,y), that is:

[0098]

[0099] In formula (1), QP represents the quantization parameter value corresponding to the first image component of the original image, x represents the abscissa value of the position of each pixel in the first image component of the original image, and y represents the ordinate value of the position of each pixel in the first image component of the original image; QP max represents the maximum value of the quantization parameter. Generally speaking, QP max takes the value of 51. However, QP max can also be other values, such as 29, 31, etc. The embodiments of the present application do not make specific limitations.

[0100] In some embodiments, before respectively obtaining at least two image components of the image to be filtered, the method further includes:

[0101] Based on at least two original image components of the image to be filtered, respectively determine the second auxiliary information corresponding to each original image component; wherein, the second auxiliary information is different from the first auxiliary information.

[0102] Optionally, in some embodiments, the determining the fusion information of the image to be filtered includes:

[0103] Perform a fusion process on at least two image components of the image to be filtered and the second auxiliary information corresponding to each original image component to obtain the fusion information of the image to be filtered.

[0104] Optionally, in some embodiments, the respectively obtaining at least two image components of the image to be filtered based on the image to be filtered includes:

[0105] Based on at least two original image components of the image to be filtered, respectively perform an addition process on each original image component in the at least two original image components and the second auxiliary information corresponding to each original image component, and after the process, obtain at least two image components of the image to be filtered.

[0106] It should be noted that the first auxiliary information and the second auxiliary information are different. Both the first auxiliary information and the second auxiliary information can be used to assist filtering and improve the filtering quality. In the embodiments of the present application, the first auxiliary information may be one or more of block partitioning information, quantization parameter information, MV information, prediction direction information, etc., and the second auxiliary information may also be one or more of block partitioning information, quantization parameter information, MV information, prediction direction information, etc.; that is to say, when the first auxiliary information is block partitioning information, the second auxiliary information may be quantization parameter information; or, when the first auxiliary information is quantization parameter information, the second auxiliary information may be block partitioning information; or, when the first auxiliary information is block partitioning information and quantization parameter information, the second auxiliary information may be MV information; or, when the first auxiliary information is block partitioning information, the second auxiliary information may be quantization parameter information and MV information; the embodiments of the present application do not make specific limitations.

[0107] It should also be noted that the fusion stages of the first auxiliary information and the second auxiliary information may be the same or different. In the embodiments of the present application, assuming that the first splitting stage is used to represent the processing stage for respectively obtaining at least two image components of the image to be filtered, the merging stage is used to represent the processing stage for determining the fusion information of the image to be filtered, and the second splitting stage is used to represent the processing stage for respectively determining the residual information of each image component after the fusion processing. In this way, the fusion stage of the first auxiliary information may be any one of the first splitting stage, the merging stage, or the second splitting stage, and the fusion stage of the second auxiliary information may also be any one of the first splitting stage, the merging stage, or the second splitting stage; that is to say, the fusion stage of the first auxiliary information may be the first splitting stage, and the fusion stage of the second auxiliary information may be the merging stage; or, the fusion stage of the first auxiliary information may be the merging stage, and the fusion stage of the second auxiliary information may be the first splitting stage; or, the fusion stage of the first auxiliary information may be the second splitting stage, and the fusion stage of the second auxiliary information may be the merging stage; or, the fusion stage of the first auxiliary information may be the first splitting stage, and the fusion stage of the second auxiliary information may be the second splitting stage; or, the fusion stage of the first auxiliary information may be the first splitting stage, and the fusion stage of the second auxiliary information may also be the first splitting stage; or,

[0108] the fusion stage of the first auxiliary information may be the merging stage, and the fusion stage of the second auxiliary information may also be the merging stage; the embodiments of the present application do not make specific limitations.

[0109] In some embodiments, performing loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component of the filtered image to be filtered includes:

[0110] Based on the fused information, at least one piece of residual information corresponding to at least one of the at least two original image components is obtained respectively;

[0111] The at least one original image component among the at least two original image components and the at least one piece of residual information corresponding to the at least one original image component are subjected to a summation operation to obtain at least one image component after filtering of the image to be filtered.

[0112] It should be noted that the loop filtering implementation method in the embodiments of the present application adopts a multi-stage cascaded processing structure, such as a split-merge-split processing structure, a split-merge processing structure, or a merge-split processing structure, etc., which are not specifically limited in the embodiments of the present application.

[0113] Specifically, if it is first necessary to separately obtain at least two image components of the image to be filtered, that is, the first split stage, and then fuse the at least two image components, that is, the merge stage; in this way, after all information fusion processing, when it is necessary to output multiple image components simultaneously, such as the first image component, the second image component, and the third image component; at this time, by jointly processing the fused information, the residual information corresponding to the first image component, the residual information corresponding to the second image component, and the residual information corresponding to the third image component are obtained respectively, and then the first image component and the residual information corresponding to the first image component are subjected to a summation operation, the second image component and the residual information corresponding to the second image component are subjected to a summation operation, and the third image component and the residual information corresponding to the third image component are subjected to a summation operation, to obtain the first image component after filtering of the image to be filtered, the second image component after filtering of the image to be filtered, and the third image component after filtering of the image to be filtered respectively, and this processing process is the second split stage; then the entire loop filtering process adopts a split-merge-split processing structure;

[0114] If it is first necessary to separately obtain at least two image components of the image to be filtered, that is, the first split stage, and then fuse the at least two image components, that is, the merge stage; in this way, after all information fusion processing, when only one image component needs to be output, such as the first image component; at this time, by jointly processing the fused information, the residual information corresponding to the first image component is obtained, and then the first image component and the residual information corresponding to the first image component are subjected to a summation operation to obtain the first image component after filtering of the image to be filtered, and this processing process does not have a second split stage; then the entire loop filtering process adopts a split-merge processing structure;

[0115] In addition, if it is not necessary to separately obtain at least two image components of the image to be filtered, that is, without the first splitting stage, the at least two image components of the image to be filtered can be directly subjected to a fusion process, that is, directly enter the merging stage; and after all the information fusion processes, since multiple image components need to be output simultaneously, a second splitting stage is also required at this time; then the entire loop filtering process adopts a merge-splitting processing structure.

[0116] It should also be noted that the loop filtering implementation method of the embodiments of the present application can also adopt more cascade processing structures, such as a splitting-merging-splitting-merging-splitting processing structure, etc. For these cascade processing structures, the embodiments of the present application can adopt a typical cascade structure, such as a splitting-merging-splitting processing structure, or can also adopt a cascade processing structure with fewer cascades than the typical cascade structure, such as a splitting-merging processing structure or a merging-splitting processing structure, etc.; even a cascade processing structure with more cascades than the typical cascade structure can be adopted, such as a splitting-merging-splitting-merging-splitting processing structure, etc., which is not specifically limited in the embodiments of the present application.

[0117] In some embodiments, the improved loop filter includes a convolutional neural network filter.

[0118] It should be noted that the improved loop filter is used to implement the loop filtering process for the image to be filtered. Among them, the improved loop filter can be a convolutional neural network filter or a filter established by other deep learning, which is not specifically limited in the embodiments of the present application. Here, the convolutional neural network filter, also known as the CNN filter, is a type of feedforward neural network that contains convolutional calculations and has a deep structure, and is one of the representative algorithms of deep learning. The input layer of the CNN filter can process multi-dimensional data, such as the three image component (Y / U / V) channels of the original image in the video to be encoded.

[0119] See Figure , which shows a schematic diagram of the composition structure of a traditional CNN filter 50 provided by the embodiments of the present application. As ​ shown, the traditional CNN filter 50 is improved on the basis of the previous generation video coding standard H.265 / High Efficiency Video Coding (HEVC). It contains a two-layer convolutional network structure and can replace the deblocking filter and the sample adaptive compensation filter. After inputting the image to be filtered (denoted by F in ), it sequentially passes through the first-layer convolutional network F1 (assuming the size of the convolutional kernel is 3×3 and contains 64 feature maps) and the second-layer convolutional network F2 (assuming the size of the convolutional kernel is 5×5 and contains 32 feature maps), and then a residual information F3 is obtained; then the image to be filtered Fin and the residual information F3 are subjected to a summation operation, and finally the filtered image output by the conventional CNN filter 50 is obtained (denoted by F out . Among them, this convolutional network structure is also called a residual neural network, which is used to output the residual information corresponding to the image to be filtered. In this conventional CNN filter 50, three image components (Y / U / V) of the image to be filtered are processed independently, but share the same filtering network and related parameters of the filtering network.

[0120] See ​ and ​ , which shows a schematic diagram of the composition structure of another conventional CNN filter 60 provided by the embodiments of the present application; this conventional CNN filter 60 uses two filtering networks, such as ​ The shown filtering network is dedicated to outputting the first image component, such as ​ The shown filtering network is dedicated to outputting the second image component or the third image component. Assume that the height of the original image in the video to be encoded is H and the width is W. Then the size information corresponding to the first image component is H×W, and the first image component can be subjected to pixel rearrangement processing to convert it into form; since the size information corresponding to the second image component or the third image component is both Then, after merging these three image components, it is transformed into form and input into the conventional CNN filter 60. Based on the filtering network such as ​ , after the input layer network receives the image F to be filtered in (assuming the size of the convolution kernel is N×N and the number of channels is 6), after passing through the first layer of convolutional network F 1-Y (assuming the size of the convolution kernel is L1×L1, the number of convolution kernels is M, and the number of channels is 6) and the second layer of convolutional network F 2-Y (assuming the size of the convolution kernel is L2×L2, the number of convolution kernels is 4, and the number of channels is M), a residual information F 3-Y (assuming the size of the convolution kernel is N×N and the number of channels is 4) is obtained; then the input image F to be filtered in and the residual information F 3-Y are subjected to a summation operation, and finally the filtered first image component output by the conventional CNN filter 60 is obtained (denoted by F out-Y . Based on the filtering network such as ​ , after the input layer network receives the image F to be filtered in (assuming the size of the convolution kernel is N×N and the number of channels is 6), after passing through the first layer of convolutional network F 1-U (assuming the size of the convolution kernel is L1×L1, the number of convolution kernels is M, and the number of channels is 6) and the second layer of convolutional network F 2-UAfter assuming that the size of the convolution kernel is L2×L2, the number of convolution kernels is 2, and the number of channels is M, a residual information F is obtained. 3-U (Assume that the size of the convolution kernel is N×N and the number of channels is 2); then the input image F to be filtered in and the residual information F 3-U are subjected to a summation operation, and finally the second image component or the third image component after filtering output by the traditional CNN filter 60 is obtained (denoted by F out-U ).

[0121] Regarding ​ the traditional CNN filter 50 shown, or ​ and ​ the traditional CNN filter 60 shown, since the relationship between different image components is not considered and it is not reasonable to independently process each image component; in addition, encoding parameters such as block partition information and QP information are not fully utilized at the input end. However, the distortion of the reconstructed image mainly comes from the blocking effect, and the boundary information of the blocking effect is determined by the CU partition information; that is to say, the filtering network in the CNN filter should focus on the boundary region; in addition, integrating the quantization parameter information into the filtering network helps to improve its generalization ability so that it can filter distorted images of any quality. Therefore, the loop filtering implementation method provided in the embodiments of the present application not only has a reasonable CNN filtering structure, and the same filtering network can receive multiple image components at the same time, but also fully considers the relationship between these multiple image components, and can simultaneously output the enhanced images of these image components after filtering processing; in addition, the loop filtering implementation method can also use encoding parameters such as block partition information and / or QP information as auxiliary information for auxiliary filtering, thereby improving the filtering quality.

[0122] It should be noted that "obtaining at least two image components of the image to be filtered" in the embodiments of the present application, specifically, it can be to determine the auxiliary information (such as the first auxiliary information or the second auxiliary information) corresponding to each image component for the first image component, the second image component, and the third image component of the image to be filtered respectively, and three image components can be obtained after the addition process; it can also be to determine the auxiliary information corresponding to each image component for the first image component and the second image component of the image to be filtered respectively, and two image components can be obtained after the addition process; it can also be to determine the auxiliary information corresponding to each image component for the first image component and the third image component of the image to be filtered respectively, and two image components can be obtained after the addition process; or even it can be to determine the auxiliary information corresponding to each image component for the second image component and the third image component of the image to be filtered respectively, and two new image components can be obtained after the addition process; the embodiments of the present application do not make specific limitations.

[0123] It should also be noted that the fusion information for the image to be filtered can be obtained by directly fusing at least two image components, or can be obtained by jointly fusing at least two image components and corresponding auxiliary information (such as the first auxiliary information or the second auxiliary information); the embodiments of the present application do not make specific limitations.

[0124] If the fusion information is obtained by directly fusing at least two image components, then it can be to fuse the first image component, the second image component, and the third image component of the image to be filtered to obtain the fusion information; it can also be to fuse the first image component and the second image component of the image to be filtered to obtain the fusion information; it can also be to fuse the first image component and the third image component of the image to be filtered to obtain the fusion information; even it can be to fuse the second image component and the third image component of the image to be filtered to obtain the fusion information.

[0125] If the fusion information is obtained by jointly fusing at least two image components and corresponding auxiliary information (such as the first auxiliary information or the second auxiliary information), then it can be to fuse the first image component, the second image component, the third image component of the image to be filtered and the auxiliary information to obtain the fusion information; it can also be to fuse the first image component, the second image component and the auxiliary information of the image to be filtered to obtain the fusion information; it can also be to fuse the first image component, the third image component and the auxiliary information of the image to be filtered to obtain the fusion information; even it can be to fuse the second image component, the third image component and the auxiliary information of the image to be filtered to obtain the fusion information. Specifically, for "obtained by jointly fusing at least two image components and corresponding auxiliary information (such as the first auxiliary information or the second auxiliary information)", it can be to first fuse at least two image components of the image to be filtered, and then incorporate the auxiliary information; it can also be to first incorporate each of the at least two image components of the image to be filtered with the corresponding auxiliary information, and then fuse the processed at least two image components; that is to say, for the specific manner of the fusion process, the embodiments of the present application do not make specific limitations.

[0126] In addition, for the step of "performing loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the filtered image to be filtered" in the embodiments of the present application, specifically, after multiple image components of the image to be filtered (such as a first image component, a second image component, and a third image component) and auxiliary information (such as first auxiliary information or second auxiliary information) are fused and input into the filtering network, it may be that only the first image component of the filtered image to be filtered, or the second image component of the filtered image, or the third image component of the filtered image is output, or it may be that the first image component of the filtered image to be filtered and the second image component of the filtered image, or the second image component of the filtered image and the third image component of the filtered image, or the first image component of the filtered image and the third image component of the filtered image are output, or even it may be the first image component of the filtered image to be filtered, the second image component of the filtered image, and the third image component of the filtered image; the embodiments of the present application do not make specific limitations.

[0127] Taking the example that three image components of the image to be filtered are simultaneously input into the filtering network and a cascade processing structure of splitting-merging-splitting is adopted, see ​ , which shows a schematic diagram of the composition structure of a loop filtering framework 70 provided by the embodiments of the present application. As ​As shown, the loop filter framework 70 may include three image components of the image to be filtered (represented by Y, U, and V respectively) 701, a first shunt unit 702, a first auxiliary information 703, a first processing unit 704 for the Y image component, a first processing unit 705 for the U image component, a first processing unit 706 for the V image component, a second auxiliary information 707, an input fusion unit 708, a joint processing unit 709, a second shunt unit 710, a second processing unit 711 for the Y image component, a second processing unit 712 for the U image component, a second processing unit 713 for the V image component, a first adder 714, a second adder 715, a third adder 716, and three filtered image components (represented by Out_Y, Out_U, and Out_V respectively) 717. Specifically, after the three image components 701 of the image to be filtered pass through the first shunt unit 702, they will be divided into three paths of signals: the Y image component, the U image component, and the V image component. The Y image component of the first path and the corresponding first auxiliary information 703 enter the first processing unit 704 for the Y image component, the U image component of the second path and the corresponding first auxiliary information 703 enter the first processing unit 705 for the U image component, and the V image component of the third path and the corresponding first auxiliary information 703 enter the first processing unit 706 for the V image component, thus outputting three new image components. The input fusion unit 708 is used to fuse these three new image components and the second auxiliary information 707, and then input them into the joint processing unit 709. The joint processing unit 709 includes a multi-layer convolutional filter network for performing convolutional calculations on the input information. Since the specific convolutional calculation process is similar to the related technical solutions, the specific implementation steps of the joint processing unit 709 will not be described here. After passing through the joint processing unit 709, it will enter the second shunt unit 710 to be re-divided into three paths of signals, and then these three paths of signals are respectively input into the second processing unit 711 for the Y image component, the second processing unit 712 for the U image component, and the second processing unit 713 for the V image component, and the residual information of the Y image component, the residual information of the U image component, and the residual information of the V image component can be obtained in sequence. The Y image component in the three image components 701 of the image to be filtered and the obtained residual information of the Y image component are jointly input into the first adder 714, and the output of the first adder 714 is the filtered Y image component (represented by Out_Y); the U image component in the three image components 701 of the image to be filtered and the obtained residual information of the U image component are jointly input into the second adder 715, and the output of the second adder 715 is the filtered U image component (represented by Out_U); the V image component in the three image components 701 of the image to be filtered and the obtained residual information of the V image component are jointly input into the third adder 716, and the output of the third adder 716 is the filtered V image component (represented by Out_V).Here, for the output component, when only the filtered Y image component needs to be output, the loop filtering framework 70 may not include the second splitting unit 710, the second adder 715, and the third adder 716; when only the filtered U image component needs to be output, the loop filtering framework 70 may not include the second splitting unit 710, the first adder 714, and the third adder 716; when the filtered Y image component and the filtered U image component need to be output, the loop filtering framework 70 may not include the third adder 716; the embodiments of the present application do not make specific limitations.

[0128] Taking the example that two image components of the image to be filtered are input into the filtering network simultaneously and a cascaded processing structure of splitting - merging is adopted, refer to ​ , which shows a schematic diagram of the composition structure of another loop filtering framework 80 provided by the embodiments of the present application. As ​ shown, the loop filtering framework 80 may include two image components (represented by Y and U respectively) 801 of the image to be filtered, a first splitting unit 702, a first auxiliary information 703, a first processing unit 704 for the Y image component, a first processing unit 705 for the U image component, an input fusion unit 708, a joint processing unit 709, a second processing unit 711 for the Y image component, a first adder 714, and one filtered image component (represented by Out_Y) 802. Specifically, after the two image components 801 of the image to be filtered pass through the first splitting unit 702, they will be divided into two paths of signals: the Y image component and the U image component. The Y image component of the first path and the corresponding first auxiliary information 703 enter the first processing unit 704 for the Y image component, and the U image component of the second path and the corresponding first auxiliary information 703 enter the first processing unit 705 for the U image component, thus outputting two new image components; the input fusion unit 708 is used to fuse these two new image components and then input them into the joint processing unit 709; after passing through the joint processing unit 709, since only a single image component (i.e., the filtered Y image component) needs to be output, it does not need to enter the second splitting unit 710 at this time and can directly enter the second processing unit 711 for the Y image component, and then the residual information of the Y image component is obtained; the Y image component in the two image components 801 of the image to be filtered and the obtained residual information of the Y image component are jointly input into the first adder 714, and the output of the first adder 714 is the filtered Y image component (represented by Out_Y).

[0129] It should be noted that since the size information of the Y image component and the U image component or the V image component may be different, in ​ the loop filtering framework 70 shown or ​In the loop filter framework 80 shown, an upsampling unit (or a deconvolution unit or a super-resolution unit) can also be added before the first processing unit 705 of the U image component and the first processing unit 706 of the V image component to perform upsampling processing, so that the resolution of the upsampled U image component or the upsampled V image component is the same as that of the Y image component, facilitating subsequent loop filter processing. Additionally, taking ​ the loop filter framework 70 shown as an example, the improved loop filter in the embodiments of the present application can at least include an input fusion unit 708, a joint processing unit 709, and a first adder 714, a second adder 715, and a third adder 716. However, it can also include a first splitting unit 702, a first processing unit 704 of the Y image component, a first processing unit 705 of the U image component, a first processing unit 706 of the V image component, etc. It can even include a second splitting unit 710, a second processing unit 711 of the Y image component, a second processing unit 712 of the U image component, a second processing unit 713 of the V image component, etc. The embodiments of the present application do not make specific limitations in this regard.

[0130] In addition, in the embodiments of the present application, the loop filter implementation method can adopt a split-merge-split processing structure, such as ​ the loop filter framework 70 shown; it can also adopt a less split-merge processing structure, such as ​ the loop filter framework 80 shown; it can also adopt a less merge-split processing structure, or even a less merge-split processing structure or a more split-merge-split-merge-split processing structure. The embodiments of the present application do not make specific limitations in this regard.

[0131] It should also be noted that the first auxiliary information and the second auxiliary information can all participate in the loop filter processing, such as ​ the loop filter framework 70 shown; the first auxiliary information and the second auxiliary information can also selectively participate in the loop filter processing, such as ​ the loop filter framework 80 shown, where the second auxiliary information does not participate in the loop filter processing. In the embodiments of the present application, it can be that both the first auxiliary information and the second auxiliary information participate in the loop filter processing, or the first auxiliary information does not participate in the loop filter processing, or the second auxiliary information does not participate in the loop filter processing, or even neither the first auxiliary information nor the second auxiliary information participates in the loop filter processing. The embodiments of the present application do not make specific limitations in this regard.

[0132] It should also be noted that the fusion stages of the first auxiliary information and the second auxiliary information may be the same or different; that is, the first auxiliary information and the second auxiliary information may participate in the loop filtering process in the same stage or in different stages, and the embodiments of the present application do not make specific limitations. For example, still taking ​ the loop filtering framework 70 shown in FIG. as an example, both the first auxiliary information 703 and the second auxiliary information 707 may participate in the loop filtering process within the stage corresponding to the first branch unit 702, or both the first auxiliary information 703 and the second auxiliary information 707 may participate in the loop filtering process within the stage corresponding to the input fusion unit 708, or both the first auxiliary information 703 and the second auxiliary information 707 may participate in the loop filtering process within the stage corresponding to the second branch unit 710; or the first auxiliary information 703 participates in the loop filtering process within the stage corresponding to the first branch unit 702, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the input fusion unit 708; or the first auxiliary information 703 participates in the loop filtering process before the stage corresponding to the first branch unit 702, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the input fusion unit 708; or the first auxiliary information 703 participates in the loop filtering process before the stage corresponding to the first branch unit 702, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the second branch unit 710; or the first auxiliary information 703 participates in the loop filtering process within the stage corresponding to the input fusion unit 708, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the second branch unit 710; that is, the first auxiliary information 703 and the second auxiliary information 707 can flexibly select the fusion stage in the cascade processing structure, and the embodiments of the present application do not make specific limitations.

[0133] Taking ​Taking the loop filter framework 70 shown as an example, it uses a deep learning network (such as a CNN) for loop filtering. The difference from a traditional CNN filter is that the improved loop filter in the embodiments of the present application adopts a cascade processing structure, which can input three image components of the image to be filtered into the filtering network simultaneously, and also incorporates other coding-related auxiliary information (such as coding parameters like block partition information, quantization parameter information, MV information, etc.). Moreover, these auxiliary information can be incorporated into the filtering network in the same stage or different stages. In this way, not only the relationship between the three image components is fully utilized, but also other coding-related auxiliary information is used to assist in filtering, improving the filtering quality. Additionally, by processing the three image components simultaneously, the problem of needing to perform three complete network forward calculations for these three image components is effectively avoided, thereby reducing the computational complexity and saving the coding bit rate. For example, taking VTM 3.0 as a reference, in a certain experimental test, it is found that compared with the related technical solutions, the loop filtering implementation method of the embodiments of the present application can achieve a 6.4% bit rate reduction for the Y image component, a 9.8% bit rate reduction for the U image component, and an 11.6% bit rate reduction for the V image component under the premise of the same restored video quality, thus saving the coding bit rate.

[0134] The above embodiments provide a loop filtering implementation method. By obtaining the image to be filtered, where the image to be filtered is generated during the video coding process of the original image in the video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image; based on the image to be filtered, at least two image components of the image to be filtered are respectively obtained; the fusion information of the image to be filtered is determined, and the fusion information at least includes that obtained by fusing the at least two image components; based on the fusion information, loop filtering processing is performed on the image to be filtered to obtain at least one image component of the filtered image to be filtered. In this way, by adopting a cascade processing structure and fusing coding parameters such as block partition information and / or QP information as auxiliary information with the input multiple image components, not only the relationship between the multiple image components is fully utilized, but also the computational complexity is reduced and the coding bit rate is saved; at the same time, the subjective and objective quality of the video reconstructed image in the encoding and decoding process is further improved.

[0135] Based on the same inventive concept as the foregoing embodiments, see ​ , which shows a schematic structural diagram of a loop filtering implementation device 90 provided by the embodiments of the present application. The loop filtering implementation device 90 may include: an acquisition unit 901, a splitting unit 902, a determination unit 903, and a filtering unit 904, where,

[0136] The obtaining unit 901 is configured to obtain an image to be filtered; wherein, the image to be filtered is generated during video encoding of an original image in a video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image;

[0137] The splitting unit 902 is configured to respectively obtain at least two image components of the image to be filtered based on the image to be filtered;

[0138] The determining unit 903 is configured to determine fusion information of the image to be filtered; wherein, the fusion information at least includes that obtained by fusing the at least two image components;

[0139] The filtering unit 904 is configured to perform loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component after filtering the image to be filtered.

[0140] In the above solution, the splitting unit 902 is specifically configured to respectively perform component processing on each of the at least two original image components in the at least two original image components of the image to be filtered, and after processing, obtain at least two image components of the image to be filtered.

[0141] In the above solution, the loop filtering implementation device 90 further includes a fusion unit 905 configured to perform fusion processing on at least two image components of the image to be filtered to obtain the fusion information of the image to be filtered.

[0142] In the above solution, the determining unit 903 is further configured to respectively determine first auxiliary information corresponding to each original image component based on at least two original image components of the image to be filtered; wherein, the first auxiliary information at least includes block partitioning information and / or quantization parameter information.

[0143] In the above solution, the splitting unit 902 is further configured to respectively perform addition processing on each of the at least two original image components in the at least two original image components of the image to be filtered and the first auxiliary information corresponding to each original image component, and after processing, obtain at least two image components of the image to be filtered.

[0144] In the above solution, the fusion unit 905 is further configured to perform fusion processing on at least two image components of the image to be filtered and the first auxiliary information corresponding to each original image component to obtain the fusion information of the image to be filtered.

[0145] In the above solution, the splitting unit 902 is further configured to respectively obtain residual information corresponding to at least one of the at least two original image components based on the fusion information;

[0146] The filtering unit 904 is specifically configured to perform a summation operation on at least one of the at least two original image components and the residual information corresponding to the at least one original image component to obtain at least one image component of the filtered image to be filtered.

[0147] In the above solution, the determining unit 903 is further configured to respectively determine second auxiliary information corresponding to each original image component based on at least two original image components of the image to be filtered; wherein the second auxiliary information is different from the first auxiliary information.

[0148] In the above solution, the fusion unit 905 is further configured to perform a fusion process on at least two image components of the image to be filtered and the second auxiliary information corresponding to each original image component to obtain the fusion information of the image to be filtered.

[0149] In the above solution, the splitting unit 902 is further configured to respectively perform an addition process on each of the at least two original image components and the second auxiliary information corresponding to each original image component based on at least two original image components of the image to be filtered, and after the process, obtain at least two image components of the image to be filtered.

[0150] In the above solution, the obtaining unit 901 is specifically configured to perform video encoding processing on the original image in the video to be encoded, and use the generated reconstructed image as the image to be filtered; or,

[0151] The obtaining unit 901 is specifically configured to perform video encoding processing on the original image in the video to be encoded to generate a reconstructed image; perform a preset filtering process on the reconstructed image, and use the preset filtered image as the image to be filtered.

[0152] In the above solution, referring to ​ , the loop filtering implementation device 90 further includes a sampling unit 906, configured to select low-resolution image components for at least two original image components of the image to be filtered; and perform upsampling processing on the low-resolution image components.

[0153] Understandably, in this embodiment, a "unit" may be a partial circuit, a partial processor, a partial program or software, etc. Of course, it may also be a module or non-modular. Moreover, the components in this embodiment may be integrated in a processing unit, may exist separately physically for each unit, or two or more units may be integrated in one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional module.

[0154] If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this embodiment essentially or the part that contributes to the prior art or all or part of this technical solution may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0155] Therefore, this embodiment provides a computer storage medium that stores a loop filtering implementation program. When the loop filtering implementation program is executed by at least one processor, the steps of the method described in the foregoing embodiment are implemented.

[0156] Based on the composition of the above loop filtering implementation device 90 and the computer storage medium, refer to ​ , which shows a specific hardware structure example of the loop filtering implementation device 90 provided by the embodiment of the present application and may include: a network interface 1001, a memory 1002, and a processor 1003; each component is coupled together through a bus system 1004. It can be understood that the bus system 1004 is used to realize the connection and communication between these components. In addition to including a data bus, the bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear description, in ​ all kinds of buses are labeled as the bus system 1004. Among them, the network interface 1001 is used for receiving and sending signals during the process of receiving and sending information with other external network elements;

[0157] The memory 1002 is used to store a computer program that can run on the processor 1003;

[0158] A processor 1003, configured to execute, when running the computer program:

[0159] Obtain an image to be filtered; wherein, the image to be filtered is generated during video encoding of an original image in a video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image;

[0160] Based on the image to be filtered, respectively obtain at least two image components of the image to be filtered;

[0161] Determine fusion information of the image to be filtered; wherein, the fusion information at least includes that obtained by fusing the at least two image components;

[0162] Based on the fusion information, perform loop filtering processing on the image to be filtered to obtain at least one image component after filtering the image to be filtered.

[0163] It can be understood that the memory 1002 in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The memory 1002 of the systems and methods described herein is intended to include but not be limited to these and any other suitable types of memories.

[0164] The processor 1003 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 1003 or the instructions in the form of software. The above-mentioned processor 1003 may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 1002, and the processor 1003 reads the information in the memory 1002 and combines its hardware to complete the steps of the above method.

[0165] It can be understood that these embodiments described herein can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For a hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, other electronic units for performing the functions described in this application, or a combination thereof.

[0166] For a software implementation, the technologies described herein can be implemented by modules (e.g., procedures, functions, etc.) that execute the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented within the processor or external to the processor.

[0167] Optionally, as another embodiment, the processor 1003 is further configured to execute the steps of the method in the foregoing embodiment when running the computer program.

[0168] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0169] The above are only the specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claimed rights.

[0170] ​

[0171] In the embodiment of the present application, first, an image to be filtered is obtained, and the image to be filtered is generated during the video encoding process of the original image in the video to be encoded; then, based on the image to be filtered, at least two image components of the image to be filtered are respectively obtained; this processing process can be regarded as a splitting stage; then, the fusion information of the image to be filtered is determined; wherein, the fusion information at least includes that obtained by fusing the at least two image components; this processing process can be regarded as a merging stage; in this way, by adopting a cascaded processing structure and fusing the input multiple image components, not only the relationship between the multiple image components is fully utilized, but also the problem of requiring multiple complete network forward calculations for the multiple image components is effectively avoided, thereby reducing the computational complexity and saving the coding bit rate; finally, based on the fusion information, loop filtering processing is performed on the image to be filtered to obtain at least one image component after filtering the image to be filtered; in this way, the fusion information can further assist in filtering, and the subjective and objective quality of the video reconstructed image in the encoding and decoding process is improved.

Claims

1. A method for implementing loop filtering, applied to a decoding system, characterized in that, The method includes: Based on at least two image components of the image to be filtered, perform component processing on the at least two image components respectively to obtain at least two image components after component processing; Perform fusion processing on the at least two image components after component processing; Based on the fusion processing, perform loop filtering processing to obtain at least one image component after filtering the image to be filtered; Wherein, the at least two image components include a first image component and a second image component, and the at least one image component after filtering the image to be filtered includes the second image component after filtering.

2. The method according to claim 1, characterized in that The performing of the fusion processing includes inputting the at least two image components after component processing at one time.

3. The method according to claim 1, characterized in that, The method further includes: Based on at least two image components of the image to be filtered, determine first auxiliary information corresponding to the at least two image components respectively; wherein, the first auxiliary information at least includes block partitioning information and / or quantization parameter information.

4. The method according to claim 3, characterized in that, The based on at least two image components of the image to be filtered, performing component processing on the at least two image components respectively to obtain at least two image components after component processing includes: Based on at least two image components of the image to be filtered, perform addition processing on the at least two image components and the corresponding first auxiliary information respectively, and after processing, obtain at least two image components of the image to be filtered.

5. The method according to claim 3, characterized in that, The performing of the fusion processing on the at least two image components after component processing includes: Perform fusion processing on the at least two image components of the image to be filtered and the corresponding first auxiliary information to obtain the fusion information of the image to be filtered.

6. The method according to claim 1, wherein The based on the fusion processing, performing loop filtering processing to obtain at least one image component after filtering the image to be filtered includes: Based on the fusion processing, respectively obtain residual information corresponding to at least one image component among the at least two image components of the image to be filtered; Perform a summation operation on at least one image component among the at least two image components of the image to be filtered and the residual information corresponding to the at least one image component to obtain at least one image component after filtering the image to be filtered.

7. The method according to claim 1, characterized in that, The method further includes: Obtain an image to be filtered, wherein the image to be filtered includes a reconstructed image, or the image to be filtered includes a reconstructed image after preset filtering processing.

8. A method for implementing loop filtering, applied to an encoding system, characterized in that, The method includes: Based on at least two image components of the image to be filtered, perform component processing on the at least two image components respectively to obtain at least two image components after component processing; Perform fusion processing on the at least two image components after component processing; Based on the fusion processing, perform loop filtering processing to obtain at least one image component after filtering the image to be filtered; Wherein, the at least two image components include a first image component and a second image component, and the at least one image component after filtering the image to be filtered includes the second image component after filtering.

9. The method according to claim 8, characterized in that, The performing of the fusion processing includes inputting the at least two image components after component processing at one time.

10. The method according to claim 8, wherein The method further includes: Based on at least two image components of the image to be filtered, respectively determine first auxiliary information corresponding to the at least two image components; wherein, the first auxiliary information at least includes block partitioning information and / or quantization parameter information.

11. The method according to claim 10, wherein The performing component processing on the at least two image components respectively based on the at least two image components of the image to be filtered to obtain at least two image components after component processing includes: Based on the at least two image components of the image to be filtered, respectively perform addition processing on the at least two image components and the corresponding first auxiliary information, and after processing, obtain at least two image components of the image to be filtered.

12. The method according to claim 10, characterized in that, The performing fusion processing on the at least two image components after component processing includes: Perform fusion processing on the at least two image components of the image to be filtered and the corresponding first auxiliary information to obtain fusion information of the image to be filtered.

13. The method according to claim 8, wherein The performing loop filtering processing based on the fusion processing to obtain at least one image component after filtering the image to be filtered includes: Based on the fusion processing, respectively obtain residual information corresponding to at least one image component among the at least two image components of the image to be filtered; Perform summation operation on at least one image component among the at least two image components of the image to be filtered and the residual information corresponding to the at least one image component to obtain at least one image component after filtering the image to be filtered.

14. The method according to claim 8, characterized in that, The method further includes: Obtain an image to be filtered, wherein the image to be filtered includes a reconstructed image, or the image to be filtered includes a reconstructed image after preset filtering processing.

15. An encoding system, the encoding system includes a first memory and a first processor, wherein: The first memory is used to store a computer program that can run on the first processor; The first processor is used to execute the method according to any one of claims 8 to 14 when running the computer program.

16. A decoding system, the decoding system includes a second memory and a second processor, wherein: The second memory is used to store a computer program that can run on the second processor; The second processor is used to execute the method according to any one of claims 1 to 7 when running the computer program.

17. A computer storage medium, characterized in that, The computer storage medium stores a loop filtering implementation program, when the loop filtering implementation program is applied to a decoding system, it implements the loop filtering method according to any one of claims 1 - 7; when the loop filtering implementation program is applied to an encoding system, it implements the loop filtering method according to any one of claims 8 - 14 and outputs a corresponding bitstream.

18. A computer storage medium, characterized in that, The computer storage medium stores a bitstream generated by the encoding method according to any one of claims 8 to 14.

Images