Loop filtering implementation method and device and computer storage medium

Abstract

The embodiment of the invention discloses a loop filtering implementation method and device and a computer storage medium. The method comprises the steps of obtaining a to-be-filtered image; wherein the to-be-filtered image is generated by an original image in a to-be-coded video in a video coding process, the to-be-coded video comprises an original image frame, and the original image frame comprises the original image; respectively obtaining at least two image components of the to-be-filtered image based on the to-be-filtered image; determining fusion information of the to-be-filtered image; wherein the fusion information at least comprises fusion of the at least two image components; and performing loop filtering processing on the to-be-filtered image based on the fusion information to obtain at least one image component of the to-be-filtered image after filtering.

Description

[0001] Description of the case

[0002] This application is a divisional application based on the Chinese patent application with application date of March 7, 2019, application number 201980090702.4, and invention name “Loop Filtering Implementation Method, Device and Computer Storage Medium”. Technical Field

[0003] The embodiments of the present application relate to the field of image processing technology, and in particular to a loop filtering implementation method, device, and computer storage medium. Background Art

[0004] In video coding and decoding systems, most video coding uses a hybrid coding framework based on block-shaped coding units (CUs). Since adjacent CUs use different coding parameters, such as different transformation processes, different quantization parameters (QPs), different prediction methods, different reference image frames, etc., and the error size and distribution characteristics introduced by each CU are independent of each other, the discontinuity of the boundaries between adjacent CUs produces blocking effects, which affects the subjective and objective quality of the reconstructed image and even affects the prediction accuracy of subsequent coding and decoding.

[0005] Therefore, during the encoding and decoding process, loop filters are used to improve the subjective and objective quality of the reconstructed image. Traditional loop filters typically artificially summarize the characteristics of distorted images, manually design the filter structure, and configure the filter coefficients. For example, deblocking filters, sample adaptive compensation, and adaptive loop filters rely on manual design. These filters do not fit the optimal filter well, and their adaptability and filtering effects are poor. In addition, the encoder needs to write the filter parameters related to local statistical information into the bitstream to ensure consistency between the encoder and decoder, which increases the number of encoded bits.

[0006] With the rapid development of deep learning theory, the industry has proposed the use of convolutional neural networks (CNNs) to filter reconstructed images to remove image distortion, achieving significant improvements in subjective and objective quality compared to traditional loop filters. However, current CNN filters do not fully utilize relevant information, resulting in limited improvements in the subjective and objective quality of reconstructed images. Summary of the Invention

[0007] The embodiments of the present application provide a loop filtering implementation method, device, and computer storage medium. By adopting a cascade processing structure and integrating coding parameters such as block partitioning information and / or QP information as auxiliary information with multiple input image components, not only the relationship between multiple image components is fully utilized, but also the computational complexity is reduced and the encoding bit rate is saved; at the same time, the subjective and objective quality of the video reconstructed image during the encoding and decoding process is further improved.

[0008] The technical solution of the embodiment of the present application can be implemented as follows:

[0009] In a first aspect, an embodiment of the present application provides a method for implementing loop filtering, the method comprising:

[0010] Acquire an image to be filtered; wherein the image to be filtered is generated during a video encoding process from an original image in a video to be encoded, the video to be encoded includes an original image frame, and the original image frame includes the original image;

[0011] Based on the image to be filtered, respectively obtain at least two image components of the image to be filtered;

[0012] Determining fusion information of the image to be filtered; wherein the fusion information at least includes information obtained by fusing the at least two image components;

[0013] Perform loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered.

[0014] In a second aspect, an embodiment of the present application provides a loop filtering implementation device, the loop filtering implementation device comprising: an acquisition unit, a branching unit, a determination unit and a filtering unit, wherein:

[0015] The acquisition unit is configured to acquire an image to be filtered; wherein the image to be filtered is generated from an original image in a video to be encoded during a video encoding process, the video to be encoded includes an original image frame, and the original image frame includes the original image;

[0016] The branching unit is configured to obtain at least two image components of the image to be filtered based on the image to be filtered;

[0017] The determining unit is configured to determine fusion information of the image to be filtered; wherein the fusion information at least includes information obtained by fusing the at least two image components;

[0018] The filtering unit is configured to perform loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered after filtering.

[0019] In a third aspect, an embodiment of the present application provides a loop filtering implementation device, the loop filtering implementation device comprising: a memory and a processor, wherein:

[0020] The memory is used to store a computer program that can be run on the processor;

[0021] The processor is configured to execute the steps of the method described in the first aspect when running the computer program.

[0022] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a loop filtering implementation program, and when the loop filtering implementation program is executed by at least one processor, the steps of the method described in the first aspect are implemented.

[0023] The embodiment of the present application provides a loop filtering implementation method, device and computer storage medium, first obtaining an image to be filtered, wherein the image to be filtered is generated during the video encoding process of the original image in the video to be encoded; then, based on the image to be filtered, at least two image components of the image to be filtered are obtained respectively; this processing process can be regarded as a branching stage; then, fusion information of the image to be filtered is determined; wherein, the fusion information at least includes the information obtained by fusing the at least two image components; this processing process can be regarded as a merging stage; in this way, by adopting a cascade processing structure and performing fusion processing on multiple input image components, not only the relationship between the multiple image components is fully utilized, but also the problem of requiring multiple complete network forward calculations for these multiple image components is effectively avoided, thereby reducing the computational complexity and saving the encoding bit rate; finally, based on the fusion information, loop filtering is performed on the image to be filtered to obtain at least one image component after filtering of the image to be filtered; in this way, the fusion information can further assist filtering, thereby improving the subjective and objective quality of the video reconstructed image during the encoding and decoding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the composition structure of a traditional coding block diagram provided for related technical solutions;

[0025] Figure 2 A schematic diagram of the structure of an improved coding block diagram provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of a loop filtering implementation method provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of a block partition matrix provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the composition structure of a traditional CNN filter provided in an embodiment of the present application;

[0029] Figure 6A and Figure 6B A schematic diagram of the composition structure of another traditional CNN filter provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of the structure of a loop filtering framework provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of the structure of another loop filtering framework provided in an embodiment of the present application;

[0032] Figure 9 A schematic diagram of the structure of a loop filtering implementation device provided in an embodiment of the present application;

[0033] Figure 10 A schematic diagram of the specific hardware structure of a loop filtering implementation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0035] In video codec systems, the video to be encoded consists of raw image frames, which in turn contain the original image. These raw images undergo various processing steps, such as prediction, transformation, quantization, reconstruction, and filtering. During these processes, the processed video image may experience pixel value offsets relative to the original image, leading to visual artifacts or artifacts. Furthermore, in the block-based CU-based hybrid coding framework adopted by most video codec systems, adjacent coding blocks use different coding parameters (such as different transformation processes, different QPs, different prediction methods, and different reference image frames). The independent magnitude and distribution of errors introduced by each coding block, coupled with discontinuities at the boundaries of adjacent coding blocks, lead to blocking artifacts. These distortions not only affect the subjective and objective quality of the reconstructed image, but can also affect the prediction accuracy of subsequent codecs if the reconstructed image is used as a reference for subsequent pixel encoding, further impacting the bitrate of the video stream. Therefore, in-loop filters are often added to video codec systems to improve the subjective and objective quality of the reconstructed image.

[0036] See also Figure 1 , which shows a traditional coding frame provided by the related technical solution Figure 10 Schematic diagram of the composition structure. Figure 1 As shown, the traditional coding frame Figure 10 It may include components such as a transform and quantization unit 101, an inverse transform and inverse quantization unit 102, a prediction unit 103, a filtering unit 104, and an entropy coding unit 105. The prediction unit 103 further includes an intra-frame prediction unit 1031 and an inter-frame prediction unit 1032. A coding tree unit (CTU) may be obtained by preliminary partitioning the input original image. Further content-adaptive partitioning of a CTU may be performed to obtain a CU, which generally contains one or more coding blocks (CBs). The coding block is subjected to intra-frame prediction by the intra-frame prediction unit 1031 or inter-frame prediction by the inter-frame prediction unit 1032 to obtain residual information. The residual information is transformed by the transform and quantization unit 101, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate. After determining the prediction mode, the prediction unit 103 is further configured to provide the selected intra-frame prediction data or inter-frame prediction data to the entropy coding unit 105. In addition, the inverse transform and inverse quantization unit 102 is configured to reconstruct the coding block by reconstructing a residual block in the pixel domain. The reconstructed residual block is filtered by the filtering unit 104 to remove blocking artifacts, and then the reconstructed residual block is added to the decoded image buffer unit to generate a reconstructed reference image. The entropy coding unit 105 is configured to encode various coding parameters and quantized transform coefficients. For example, the entropy coding unit 105 uses header information coding and context-based adaptive binary arithmetic coding (Context-based Adaptive Binary Arithmatic Coding). Coding, CABAC) algorithm can be used to encode coding information indicating the determined prediction mode and output a corresponding bit stream.

[0037] against Figure 1 The traditional coding frame Figure 10The filtering unit 104 is a loop filter, also known as an in-loop filter, which may include a deblocking filter (DBF) 1041, a sample adaptive offset (SAO) filter 1042, and an adaptive loop filter (ALF) 1043. The deblocking filter 1041 is used to implement deblocking filtering. In the next-generation video coding standard H.266 / Versatile Video Coding (VVC), for all coding block boundaries in the original image, the boundary strength is first determined based on the coding parameters on both sides of the boundary, and the block boundary texture value is calculated to determine whether to perform deblocking filtering. Finally, the pixel information on both sides of the coding block boundary is corrected based on the boundary strength and filtering decision. In VVC, after deblocking filtering is performed, SAO technology, namely the sample adaptive offset filter 1042, is introduced to reduce the quantization distortion of high-frequency AC coefficients. Furthermore, starting from the pixel domain, negative values ​​are added to pixels at peaks and positive values ​​are added to pixels at troughs for compensation. In VVC, after deblocking filtering and sample adaptive offset filtering are performed, it is necessary to further use the adaptive loop filter 1043 for filtering. For adaptive loop filtering, it is based on the pixel values ​​of the original image and the pixel values ​​of the distorted image to calculate the optimal filter in the mean square sense. However, these filters (such as the deblocking filter 1041, the sample adaptive compensation filter 1042 and the adaptive loop filter 1043) not only require careful manual design and a large number of judgments and decisions; but also, at the encoding end, the filter-related parameters that depend on local statistical information (such as the filter coefficient and the flag value of whether to select the filter, etc.) need to be written into the bit stream to ensure consistency between the encoding and decoding ends, which increases the number of encoded bits; at the same time, the manually designed filters do not fit the complex functions of the actual optimization targets to a high degree, and the filtering effect needs to be enhanced.

[0038] The embodiment of the present application provides a loop filtering implementation method, which is applied to the improved coding block diagram; Figure 1 The traditional coding frame shown Figure 10Compared to the prior art, the main difference is that an improved loop filter is used to replace the deblocking filter 1041, the sample adaptive compensation filter 1042, and the adaptive loop filter 1043 in the related art. In the embodiment of the present application, the improved loop filter can be a convolutional neural network (CNN) filter or a filter established by other deep learning methods, which is not specifically limited in the embodiment of the present application.

[0039] Taking convolutional neural network filters as an example, see Figure 2 , which shows a schematic diagram of the composition structure of an improved encoding block diagram 20 provided in an embodiment of the present application. Figure 2 As shown, compared with the traditional coding frame Figure 10 In contrast, the filter unit 104 in the improved encoding block diagram 20 includes a convolutional neural network filter 201. The convolutional neural network filter 201 can not only completely replace Figure 1 The deblocking filter 1041, the sample adaptive compensation filter 1042 and the adaptive loop filter 1043 in the embodiment of the present invention can also partially replace Figure 1 Any one or two of the deblocking filter 1041, the sample adaptive compensation filter 1042 and the adaptive loop filter 1043 in the embodiment of the present invention can also be combined with Figure 1 Any one or more of the deblocking filter 1041, the sample adaptive compensation filter 1042 and the adaptive loop filter 1043 in the embodiment of the present invention are used in combination. Figure 1 or Figure 2 Each component shown in the figure, such as the transform and quantization unit 101, the inverse transform and inverse quantization unit 102, the prediction unit 103, the filtering unit 104, the entropy coding unit 105, or the convolutional neural network filter 201, can be a virtual module or a hardware module. In addition, those skilled in the art will understand that these units do not constitute a limitation of the encoding block diagram, and the encoding block diagram may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0040] In an embodiment of the present application, the convolutional neural network filter 201 can be directly deployed at the encoding end and the decoding end after the filter network training, so there is no need to transmit any filter-related parameters; and the convolutional neural network filter 201 can also fuse auxiliary information such as block division information and / or QP information with multiple input image components; in this way, not only the relationship between multiple image components is fully utilized, but also the computational complexity is reduced and the encoding bit rate is saved; at the same time, the subjective and objective quality of the video reconstructed image in the encoding and decoding process is further improved.

[0041] It should be noted that the loop filtering implementation method of the present application embodiment can be applied not only to encoding systems but also to decoding systems. Generally speaking, to save encoding bit rate while ensuring that the decoding system can perform correct decoding processing, the loop filter of the present application embodiment must be deployed simultaneously in the encoding and decoding systems. The following detailed description will be given using the application in the encoding system as an example.

[0042] See also Figure 3 , which shows a flow chart of a loop filtering implementation method provided by an embodiment of the present application, which may include:

[0043] S301: Acquire an image to be filtered; wherein the image to be filtered is generated during a video encoding process from an original image in a video to be encoded, the video to be encoded includes an original image frame, and the original image frame includes the original image;

[0044] S302: Based on the image to be filtered, respectively obtain at least two image components of the image to be filtered;

[0045] S303: Determine fusion information of the image to be filtered; wherein the fusion information at least includes information obtained by fusing the at least two image components;

[0046] S304: Perform loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered.

[0047] It should be noted that since the original image can be divided into CTUs, or the CTUs can be divided into CUs; ​​that is, the block division information in the embodiment of the present application may refer to CTU division information, or may refer to CU division information; in this way, the loop filtering implementation method of the embodiment of the present application can be applied not only to CU-level loop filtering, but also to CTU-level loop filtering, and the embodiment of the present application does not make specific limitations.

[0048] In an embodiment of the present application, after obtaining the image to be filtered, at least two image components of the image to be filtered are obtained respectively based on the image to be filtered; this processing process can be regarded as a branching stage, which is used to obtain at least two image components respectively; then the fusion information of the image to be filtered is determined; wherein, the fusion information at least includes the information obtained by fusing the at least two image components; this processing process can be regarded as a merging stage, which is used to fuse at least two image components; in this way, the embodiment of the present application adopts a cascade processing structure, which not only fully utilizes the relationship between the multiple image components by fusing the input multiple image components, but also effectively avoids the problem of needing to perform multiple complete network forward calculations on these multiple image components, thereby reducing the computational complexity and saving the encoding bit rate; finally, the image to be filtered is loop filtered based on the fusion information to obtain at least one image component after filtering of the image to be filtered; in this way, the fusion information can further assist filtering, thereby improving the subjective and objective quality of the video reconstructed image during the encoding and decoding process.

[0049] In some embodiments, the image components include a first image component, a second image component, and a third image component; wherein the first image component represents a luminance component, the second image component represents a first chrominance component, and the third image component represents a second chrominance component.

[0050] It should be noted that, in a video image, a first image component, a second image component, and a third image component are generally used to represent the original image or the image to be filtered. Among them, under the luminance-chrominance component representation method, the three image components are respectively a luminance component, a blue chrominance (color difference) component, and a red chrominance (color difference) component; specifically, the luminance component is usually represented by the symbol Y, the blue chrominance component is usually represented by the symbol Cb, and can also be represented by U; the red chrominance component is usually represented by the symbol Cr, and can also be represented by V. In the embodiment of the present application, the first image component can be the luminance component Y, the second image component can be the blue chrominance component U, and the third image component can be the red chrominance component V, but the embodiment of the present application is not specifically limited. Among them, at least one image component represents one or more of the first image component, the second image component, and the third image component, and the at least two image components can be the first image component, the second image component, and the third image component, or can be the first image component and the second image component, or can be the first image component and the third image component, or even can be the second image component and the third image component, and the embodiment of the present application is not specifically limited.

[0051] In the next generation video coding standard VVC, its corresponding test model is the VVC Test Model (VTM). When implementing the VTM test, the current standard test sequence uses the YUV 4:2:0 format. Each frame of the video to be encoded in this format can be composed of three image components: a luminance component (represented by Y) and two chrominance components (represented by U and V). Assuming that the height of the original image in the video to be encoded is H and the width is W, then the size information corresponding to the first image component is H×W, and the size information corresponding to the second or third image component is It should be noted that the embodiment of the present application will be described using YUV in 4:2:0 format as an example, but the loop filtering implementation method of the embodiment of the present application is also applicable to other sampling formats.

[0052] Taking YUV 4:2:0 format as an example, since the size information of the first image component is different from that of the second image component or the third image component, in order to input the first image component and / or the second image component and / or the third image component into the loop filter model at one time, these three image components need to be sampled or reorganized so that the spatial size information of the three image components is the same.

[0053] In some embodiments, pixel rearrangement processing (also referred to as downsampling processing) may be performed on the high-resolution image component so that the spatial size information of the three image components is the same. Specifically, before obtaining at least two image components of the image to be filtered based on the image to be filtered, the method further includes:

[0054] For at least two original image components of the image to be filtered, selecting a high-resolution image component;

[0055] A pixel rearrangement process is performed on the high-resolution image component.

[0056] It should be noted that the three image components contained in the original image (such as the first image component, the second image component and the third image component) are the original image components before other processing. If the first image component is the luminance component, the second image component is the first chrominance component, and the third image component is the second chrominance component; then the high-resolution image component is the first image component, and the first image component needs to be pixel rearranged. For example, taking the original image of size 2×2 as an example, convert it into 4 channels, that is, arrange the 2×2×1 tensor into a 1×1×4 tensor; then when the size information of the first image component of the original image is H×W, it can be converted into H×W by pixel rearrangement before loop filtering. In the form of; Since the size information of the second image component and the third image component are In this way, the spatial size information of the three image components can be made the same; subsequently, the first image component, the second image component and the third image component after the pixel rearrangement processing are merged and transformed into The improved loop filter is input in the form of .

[0057] In some embodiments, the low-resolution image component may be upsampled so that the spatial size information of the three image components is the same. Specifically, before obtaining at least two image components of the image to be filtered based on the image to be filtered, the method further includes:

[0058] For at least two original image components of the image to be filtered, selecting a low-resolution image component;

[0059] An upsampling process is performed on the low-resolution image component.

[0060] It should be noted that, in addition to performing pixel rearrangement processing (i.e., downward adjustment) of the size information of the high-resolution image component, in the embodiment of the present application, upsampling processing (i.e., upward adjustment) can also be performed on the low-resolution image component. In addition, for the low-resolution image component, not only upsampling processing can be performed, but also deconvolution processing and even super-resolution processing can be performed. The effects of these three processings are the same and are not specifically limited in the embodiment of the present application.

[0061] It should also be noted that the three image components contained in the original image (such as the first image component, the second image component and the third image component) are the original image components before other processing is performed. If the first image component is the luminance component, the second image component is the first chrominance component, and the third image component is the second chrominance component; then the low-resolution image component is the second image component or the third image component, and the second image component or the third image component needs to be upsampled. For example, when the size information of the second image component and the third image component of the original image are both When the image component is upsampled, it can be converted into an H×W format before loop filtering. Since the size information of the first image component is H×W, the spatial size information of the three image components can be the same, and the resolution of the upsampled second image component and the upsampled third image component will be consistent with that of the first image component.

[0062] In some embodiments, obtaining the image to be filtered includes:

[0063] Performing video encoding processing on the original image in the video to be encoded, and using the generated reconstructed image as the image to be filtered; or

[0064] Perform video encoding processing on the original image in the video to be encoded to generate a reconstructed image; perform preset filtering processing on the reconstructed image, and use the image after the preset filtering as the image to be filtered.

[0065] It should be noted that, based on the improved encoding block diagram 20, in the process of video encoding the original image in the video to be encoded, when the video encoding processing is performed on the original image, it is subjected to CU division, prediction, transformation and quantization, and in order to obtain a reference image for video encoding of the subsequent image to be encoded, it can also be subjected to inverse transformation and inverse quantization, reconstruction and filtering, etc. Thus, the image to be filtered in the embodiment of the present application can be a reconstructed image generated after reconstruction processing in the video encoding process, or it can be a preset filtered image obtained by pre-setting filtering the reconstructed image using other preset filtering methods (such as a deblocking filtering method), and the embodiment of the present application does not specifically limit this.

[0066] In some embodiments, before respectively obtaining at least two image components of the image to be filtered, the method further includes:

[0067] Based on at least two original image components of the image to be filtered, first auxiliary information corresponding to each original image component is determined respectively; wherein the first auxiliary information at least includes block division information and / or quantization parameter information.

[0068] It can be understood that the first auxiliary information can be used to assist filtering and improve filtering quality. In an embodiment of the present application, the first auxiliary information can be not only block partitioning information (such as CU partitioning information and / or CTU partitioning information), but also quantization parameter information, and even motion vector (MV) information, prediction direction information, etc.; this information can be used alone as the first auxiliary information, or can be arbitrarily combined as the first auxiliary information, such as using block partitioning information alone as the first auxiliary information, or using block partitioning information and quantization parameter information together as the first auxiliary information, or using block partitioning information and MV information together as the first auxiliary information, etc., which is not specifically limited in the embodiment of the present application.

[0069] Optionally, in some embodiments, obtaining at least two image components of the image to be filtered based on the image to be filtered includes:

[0070] Based on the at least two original image components of the image to be filtered, component processing is performed on each of the at least two original image components to obtain the at least two image components of the image to be filtered after processing.

[0071] Optionally, in some embodiments, obtaining at least two image components of the image to be filtered based on the image to be filtered includes:

[0072] Based on the at least two original image components of the image to be filtered, each original image component of the at least two original image components and the first auxiliary information corresponding to each original image component are added and processed, and after processing, the at least two image components of the image to be filtered are obtained.

[0073] It should be noted that "respectively obtaining at least two image components of the image to be filtered" can be considered the first branching stage. Thus, component processing (e.g., deep learning) can be performed on the at least two original image components of the image to be filtered, thereby obtaining at least two image components. Furthermore, the first auxiliary information corresponding to each original image component can be added to the corresponding image component, thereby obtaining at least two image components. In other words, the first auxiliary information may or may not be added during the first branching stage, and this is not specifically limited in this embodiment of the present application.

[0074] Optionally, in some embodiments, determining the fusion information of the image to be filtered includes:

[0075] A fusion process is performed on at least two image components of the image to be filtered to obtain fusion information of the image to be filtered.

[0076] Optionally, in some embodiments, determining the fusion information of the image to be filtered includes:

[0077] A fusion process is performed on at least two image components of the image to be filtered and the first auxiliary information corresponding to each original image component to obtain fusion information of the image to be filtered.

[0078] It should be noted that "determining the fusion information of the image to be filtered" can be considered a merging stage. Thus, the fusion information of the image to be filtered can be obtained by directly fusing at least two image components, or by fusing at least two image components and corresponding first auxiliary information. In other words, the first auxiliary information may or may not be added during the merging stage, and this is not specifically limited in this embodiment of the present application.

[0079] In related technical solutions, such as Figure 1The filter in the filtering unit 104 shown processes the three image components independently, requiring a total of three complete network forward calculations; while the loop filtering implementation method of the embodiment of the present application can input these three image components at one time by fusing information, or can input any two of the three image components at the same time, which is not specifically limited in the embodiment of the present application; this effectively avoids the problem of requiring multiple complete network forward calculations for multiple image components, thereby reducing computational complexity.

[0080] It can also be understood that since the original image can be divided into CTUs, or CTUs can be divided into CUs, the loop filtering implementation method of the embodiment of the present application can be applied not only to CU-level loop filtering (in which case the block partition information is CU partition information), but also to CTU-level loop filtering (in which case the block partition information is CTU partition information), and the embodiment of the present application does not specifically limit this. The following description will take CU partition information as the block partition information as an example.

[0081] In some embodiments, determining the first auxiliary information corresponding to each image component includes:

[0082] Based on the original image in the video to be encoded, each of the at least two image components of the original image is divided into a coding unit CU to obtain CU division information, and the CU division information is used as block division information corresponding to each image component of the image to be filtered.

[0083] Furthermore, in some embodiments, using the CU partition information as block partition information corresponding to each image component of the image to be filtered includes:

[0084] For the CU partition information, fill each pixel position corresponding to the CU boundary with a first value, and fill other pixel positions with a second value, to obtain a first matrix corresponding to the CU partition information; wherein the first value is different from the second value;

[0085] The first matrix is ​​used as block division information corresponding to each image component of the image to be filtered.

[0086] It should be noted that the first value can be a pre-set number, letter, etc., and the second value can also be a pre-set number, letter, etc., and the first value is different from the second value; for example, the first value can be set to 2, and the second value can be set to 1, but the embodiment of this application does not make specific limitations.

[0087] In the embodiment of the present application, the CU partition information can be used as the first auxiliary information to assist in filtering the image to be filtered. That is, during the video encoding process of the original image in the video to be encoded, the CU partition information can be fully utilized and fused with at least two image components of the image to be filtered to guide the filtering.

[0088] Specifically, the CU division information is converted into a coding unit map (Coding Unit Map, CUmap) and represented by a two-dimensional matrix, namely the CUmap matrix, which is also the first matrix in the embodiment of the present application; that is, taking the first image component of the original image as an example, it can be divided into multiple CUs; ​​each pixel position corresponding to each CU boundary is filled with the first value, and the other pixel positions are filled with the second value, so that a first matrix reflecting the CU division information can be constructed. For example, see Figure 4 , which shows a schematic diagram of the structure of a block partition matrix provided by an embodiment of the present application. Figure 4 As shown, if the figure represents a CTU, then the CTU can be divided into 9 CUs; ​​assuming that the first value is set to 2 and the second value is set to 1; in this way, each pixel position corresponding to the boundary of each CU is filled with 2, and the other pixel positions are filled with 1, that is, the pixel position filled with 2 represents the boundary of the CU, so that the CU division information can be determined, that is, the first auxiliary information corresponding to the first image component of the image to be filtered.

[0089] It should also be noted that if the first image component is a luminance component and the second and third image components are chrominance components, then the CU split information of the first image component may be different from the CU split information of the second or third image component. Therefore, when the CU split information of the first image component is different from the CU split information of the second or third image component, it is necessary to separately determine the CU split information corresponding to the first image component of the image to be filtered and the CU split information corresponding to the second or third image component of the image to be filtered; then add the CU split information as the first auxiliary information to the corresponding first, second, or third image component; when the CU split information of the first image component is the same as the CU split information of the second or third image component, then only the CU split information of the first, second, or third image component may be determined, and then the determined CU split information may be added as the first auxiliary information to the corresponding first, second, or third image component; in this way, it is convenient to subsequently fuse the at least two new image components obtained to perform loop filtering on the image to be filtered.

[0090] In some embodiments, determining the first auxiliary information corresponding to each image component includes:

[0091] Based on the original image in the video to be encoded, quantization parameters corresponding to each image component of at least two image components of the original image are respectively obtained, and the quantization parameters are used as quantization parameter information corresponding to each image component of the image to be filtered.

[0092] Furthermore, in some embodiments, using the quantization parameter as quantization parameter information corresponding to each image component of the image to be filtered includes:

[0093] Establishing a second matrix having the same size as each image component of the original image; wherein each pixel position in the second matrix is ​​filled with the normalized value of the quantization parameter corresponding to each image component of the original image;

[0094] The second matrix is ​​used as quantization parameter information corresponding to each image component of the image to be filtered.

[0095] It should be noted that different quantization parameters correspond to different degrees of distortion in the filtered image. If the quantization parameter information is incorporated, the filtering network can be adaptively equipped to handle any quantization parameter during training.

[0096] In an embodiment of the present application, quantization parameter information can also be used as first auxiliary information to assist in filtering the image to be filtered. That is, during the process of encoding the original image in the video to be encoded, the quantization parameter information can be fully utilized and fused with at least two image components of the image to be filtered to guide filtering. The quantization parameter information can be normalized or non-normalized (e.g., classified or partitioned); the following will describe the quantization parameter normalization process in detail.

[0097] Specifically, the quantization parameter information is converted into a second matrix reflecting the quantization parameter information; that is, taking the first image component of the original image as an example, a matrix with the same size as the first image component of the original image is established, and each pixel position in the matrix is ​​filled with the normalized value of the quantization parameter corresponding to the first image component of the original image; wherein the normalized value of the quantization parameter is denoted by QP max (x,y) means:

[0098]

[0099] In formula (1), QP represents the quantization parameter value corresponding to the first image component of the original image, x represents the horizontal coordinate value of each pixel position in the first image component of the original image, and y represents the vertical coordinate value of each pixel position in the first image component of the original image; QP max Indicates the maximum value of the quantization parameter. Generally speaking, QP max The value of is 51, but QP max It may also be other values, such as 29, 31, etc., which is not specifically limited in the embodiments of the present application.

[0100] In some embodiments, before respectively obtaining at least two image components of the image to be filtered, the method further includes:

[0101] Based on at least two original image components of the image to be filtered, second auxiliary information corresponding to each original image component is determined respectively; wherein the second auxiliary information is different from the first auxiliary information.

[0102] Optionally, in some embodiments, determining the fusion information of the image to be filtered includes:

[0103] A fusion process is performed on at least two image components of the image to be filtered and the second auxiliary information corresponding to each original image component to obtain fusion information of the image to be filtered.

[0104] Optionally, in some embodiments, obtaining at least two image components of the image to be filtered based on the image to be filtered includes:

[0105] Based on the at least two original image components of the image to be filtered, each original image component of the at least two original image components and the second auxiliary information corresponding to each original image component are added and processed, and after processing, the at least two image components of the image to be filtered are obtained.

[0106] It should be noted that the first auxiliary information and the second auxiliary information are different. Both the first auxiliary information and the second auxiliary information can be used to assist filtering and improve filtering quality. In an embodiment of the present application, the first auxiliary information may be one or more of block division information, quantization parameter information, MV information, and prediction direction information, and the second auxiliary information may also be one or more of block division information, quantization parameter information, MV information, and prediction direction information; that is, when the first auxiliary information is block division information, the second auxiliary information may be quantization parameter information; or, when the first auxiliary information is quantization parameter information, the second auxiliary information may be block division information; or, when the first auxiliary information is block division information and quantization parameter information, the second auxiliary information may be MV information; or, when the first auxiliary information is block division information, the second auxiliary information may be quantization parameter information and MV information; the embodiment of the present application does not make specific limitations.

[0107] It should also be noted that the fusion stages of the first auxiliary information and the second auxiliary information can be the same or different. In the embodiment of the present application, it is assumed that the first branching stage is used to represent the processing stage corresponding to obtaining at least two image components of the image to be filtered, the merging stage is used to represent the processing stage corresponding to determining the fusion information of the image to be filtered, and the second branching stage is used to represent the processing stage corresponding to determining the residual information of each image component after the fusion process. In this way, the fusion stage of the first auxiliary information can be any one of the first branching stage, the merging stage or the second branching stage, and the fusion stage of the second auxiliary information can also be any one of the first branching stage, the merging stage or the second branching stage; that is, the fusion stage of the first auxiliary information can be the first branching stage, and the fusion stage of the second auxiliary information can be the merging stage; or, the fusion stage of the first auxiliary information can be the merging stage, and the fusion stage of the second auxiliary information can be the first branching stage; or, the fusion stage of the first auxiliary information can be the second branching stage, and the fusion stage of the second auxiliary information can be the merging stage; or, the fusion stage of the first auxiliary information can be the first branching stage, and the fusion stage of the second auxiliary information can be the second branching stage; or, the fusion stage of the first auxiliary information can be the first branching stage, and the fusion stage of the second auxiliary information can also be the first branching stage; or, the fusion stage of the first auxiliary information can be the merging stage, and the fusion stage of the second auxiliary information can also be the merging stage; the embodiments of the present application do not make specific limitations.

[0108] In some embodiments, performing loop filtering on the image to be filtered based on the fusion information to obtain at least one filtered image component of the image to be filtered includes:

[0109] Based on the fusion information, respectively obtaining residual information corresponding to at least one original image component of the at least two original image components;

[0110] A sum operation is performed on at least one original image component of the at least two original image components and the residual information corresponding to the at least one original image component to obtain at least one image component after filtering of the image to be filtered.

[0111] It should be noted that the loop filtering implementation method of the embodiment of the present application adopts a multi-stage cascade processing structure, such as a branch-merge-branch processing structure, a branch-merge processing structure, or a merge-branch processing structure, etc., which is not specifically limited in the embodiment of the present application.

[0112] Specifically, if it is necessary to first obtain at least two image components of the image to be filtered separately, that is, the first branching stage, and then fuse the at least two image components, that is, the merging stage; in this way, after all information is fused, when multiple image components need to be output simultaneously, such as the first image component, the second image component and the third image component; at this time, by jointly processing the fused information, the residual information corresponding to the first image component, the residual information corresponding to the second image component and the residual information corresponding to the third image component are obtained respectively, and then the first image component and the residual information corresponding to the first image component are summed, the second image component and the residual information corresponding to the second image component are summed, and the third image component and the residual information corresponding to the third image component are summed, respectively obtaining the first image component after filtering of the image to be filtered, the second image component after filtering of the image to be filtered and the third image component after filtering of the image to be filtered, this processing process is the second branching stage; then the entire loop filtering process adopts a branching-merging-branching processing structure;

[0113] If it is necessary to first obtain at least two image components of the image to be filtered separately, that is, the first branching stage, and then fuse the at least two image components, that is, the merging stage; in this way, after all information is fused, when only one image component needs to be output, such as the first image component; at this time, by jointly processing the fused information to obtain the residual information corresponding to the first image component, and then summing the first image component with the residual information corresponding to the first image component to obtain the first image component after filtering of the image to be filtered, this processing process does not have a second branching stage; then the entire loop filtering process adopts a branching-merging processing structure;

[0114] In addition, if there is no need to obtain at least two image components of the image to be filtered separately, that is, there is no need for the first branching stage, the at least two image components of the image to be filtered can be directly fused, that is, directly entering the merging stage; and after all information is fused, since multiple image components need to be output at the same time, a second branching stage is still required at this time; then the entire loop filtering process adopts a merging-branching processing structure.

[0115] It should also be noted that the loop filtering implementation method of the embodiment of the present application can also adopt more cascade processing structures, such as a branch-merge-branch-merge-branch processing structure, etc. With respect to these cascade processing structures, the embodiment of the present application can adopt a typical cascade structure, such as a branch-merge-branch processing structure, and can also adopt a cascade processing structure with fewer than the typical cascade structure, such as a branch-merge processing structure or a merge-branch processing structure; it can even adopt a cascade processing structure with more than the typical cascade structure, such as a branch-merge-branch-merge-branch processing structure, etc., and the embodiment of the present application does not specifically limit this.

[0116] In some embodiments, the improved loop filter comprises a convolutional neural network filter.

[0117] It should be noted that the improved loop filter is used to implement loop filtering processing of the image to be filtered. Among them, the improved loop filter can be a convolutional neural network filter, or it can be a filter established by other deep learning, and the embodiments of the present application are not specifically limited. Here, the convolutional neural network filter, also known as the CNN filter, is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. The input layer of the CNN filter can process multidimensional data, such as the three image components (Y / U / V) channels of the original image in the video to be encoded.

[0118] See also Figure 5 , which shows a schematic diagram of the composition structure of a traditional CNN filter 50 provided in an embodiment of the present application. Figure 5 As shown, the traditional CNN filter 50 is improved on the basis of the previous generation video coding standard H.265 / High Efficiency Video Coding (HEVC), which includes a two-layer convolutional network structure and can replace the deblocking filter and the sample adaptive compensation filter. in After the image to be filtered (represented) is input into the input layer of the traditional CNN filter 50, it passes through the first convolution network F1 (assuming the size of the convolution kernel is 3×3, containing 64 feature maps) and the second convolution network F2 (assuming the size of the convolution kernel is 5×5, containing 32 feature maps) in sequence to obtain a residual information F3; then the image to be filtered Fin The summation operation is performed with the residual information F3, and finally the filtered image output by the traditional CNN filter 50 is obtained (using F out denoted by ). The convolutional network structure is also called a residual neural network, which is used to output residual information corresponding to the image to be filtered. In the traditional CNN filter 50, the three image components (Y / U / V) of the image to be filtered are processed independently, but share the same filtering network and related parameters of the filtering network.

[0119] See also Figure 6A and Figure 6B , which shows a schematic diagram of the composition structure of another traditional CNN filter 60 provided in an embodiment of the present application; the traditional CNN filter 60 uses two filter networks, such as Figure 6A The filtering network shown is dedicated to outputting the first image component, such as Figure 6B The filtering network shown is dedicated to outputting the second image component or the third image component. Assuming that the height of the original image in the video to be encoded is H and the width is W, then the size information corresponding to the first image component is H×W. The first image component can be pixel-rearranged to convert it into In the form of; since the size information corresponding to the second image component or the third image component is Then the three image components are combined and transformed into is input into the traditional CNN filter 60. Figure 6A The filtering network shown in the figure, the input layer network receives the image to be filtered F in (Assuming the size of the convolution kernel is N×N and the number of channels is 6), after the first layer of convolutional network F 1-Y (Assume that the size of the convolution kernel is L1×L1, the number of convolution kernels is M, and the number of channels is 6) and the second layer convolution network F 2-Y (Assuming the size of the convolution kernel is L2×L2, the number of convolution kernels is 4, and the number of channels is M) then a residual information F is obtained. 3-Y (Assume that the size of the convolution kernel is N×N and the number of channels is 4); then input the image to be filtered F in and residual information F 3-Y Perform summation to finally obtain the filtered first image component output by the traditional CNN filter 60 (using F out-Y Based on Figure 6B The filtering network shown in the figure, the input layer network receives the image to be filtered F in (Assuming the size of the convolution kernel is N×N and the number of channels is 6), after the first layer of convolutional network F 1-U (Assume that the size of the convolution kernel is L1×L1, the number of convolution kernels is M, and the number of channels is 6) and the second layer convolution network F 2-U(Assuming the size of the convolution kernel is L2×L2, the number of convolution kernels is 2, and the number of channels is M) then a residual information F is obtained. 3-U (Assume that the size of the convolution kernel is N×N and the number of channels is 2); then input the image to be filtered F in and residual information F 3-U Perform summation operation to finally obtain the filtered second image component or the filtered third image component output by the traditional CNN filter 60 (using F out-U express).

[0120] against Figure 5 The conventional CNN filter 50 shown, or Figure 6A and Figure 6B The traditional CNN filter 60 shown does not take into account the relationship between different image components, and it is not reasonable to process each image component independently; in addition, the block division information, QP information and other coding parameters are not fully utilized at the input end. However, the distortion of the reconstructed image mainly comes from the block effect, and the boundary information of the block effect is determined by the CU division information; that is, the filtering network in the CNN filter should focus on the boundary area; in addition, incorporating quantization parameter information into the filtering network can also help improve its generalization ability, so that it can filter distorted images of any quality. Therefore, the loop filtering implementation method provided in the embodiment of the present application not only has a reasonable CNN filtering structure, the same filtering network can receive multiple image components at the same time, and fully considers the relationship between these multiple image components, and can also output enhanced images of these image components at the same time after filtering processing; in addition, the loop filtering implementation method can also be used as auxiliary information by incorporating coding parameters such as block division information and / or QP information for auxiliary filtering, thereby improving the filtering quality.

[0121] It should be noted that, in the embodiment of the present application, “obtaining at least two image components of the image to be filtered”, specifically, can be determining the auxiliary information corresponding to each image component (such as the first auxiliary information or the second auxiliary information) for the first image component, the second image component and the third image component of the image to be filtered, respectively, and three image components can be obtained after adding processing; it can also be determining the auxiliary information corresponding to each image component for the first image component and the second image component of the image to be filtered, respectively, and two image components can be obtained after adding processing; it can also be determining the auxiliary information corresponding to each image component for the first image component and the third image component of the image to be filtered, respectively, and two image components can be obtained after adding processing; it can even be determining the auxiliary information corresponding to each image component for the second image component and the third image component of the image to be filtered, respectively, and two new image components can be obtained after adding processing; the embodiment of the present application does not make specific limitations.

[0122] It should also be noted that the fusion information of the image to be filtered can be obtained by directly fusing at least two image components, or by fusing at least two image components and corresponding auxiliary information (such as first auxiliary information or second auxiliary information); this embodiment of the present application does not make specific limitations.

[0123] If the fusion information is obtained by directly fusing at least two image components, then the first image component, the second image component and the third image component of the image to be filtered can be fused to obtain the fusion information; the first image component and the second image component of the image to be filtered can be fused to obtain the fusion information; the first image component and the third image component of the image to be filtered can be fused to obtain the fusion information; and even the second image component and the third image component of the image to be filtered can be fused to obtain the fusion information.

[0124] If the fusion information is obtained by fusing at least two image components and corresponding auxiliary information (such as the first auxiliary information or the second auxiliary information), then the first image component, the second image component, and the third image component of the image to be filtered may be fused with the auxiliary information to obtain the fusion information; the first image component, the second image component, and the auxiliary information may be fused to obtain the fusion information; the first image component, the third image component, and the auxiliary information may be fused to obtain the fusion information; or the second image component, the third image component, and the auxiliary information may be fused to obtain the fusion information. Specifically, with respect to "obtained by fusing at least two image components and corresponding auxiliary information (such as the first auxiliary information or the second auxiliary information)", the at least two image components of the image to be filtered may be first fused and then the auxiliary information may be incorporated; or each of the at least two image components of the image to be filtered may be first separately fused with the corresponding auxiliary information and then the at least two processed image components may be fused; that is, the specific method of the fusion processing is not specifically limited in the embodiments of the present application.

[0125] In addition, in the embodiment of the present application, "loop filtering is performed on the image to be filtered based on the fusion information to obtain at least one image component after filtering of the image to be filtered". Specifically, after multiple image components of the image to be filtered (such as the first image component, the second image component and the third image component) and auxiliary information (such as the first auxiliary information or the second auxiliary information) are fused and input into the filtering network, it can be output only the first filtered image component of the image to be filtered, or the second filtered image component, or the third filtered image component. It can also be output the first filtered image component and the second filtered image component of the image to be filtered, or the second filtered image component and the third filtered image component, or the first filtered image component and the third filtered image component, or even the first filtered image component, the second filtered image component and the third filtered image component of the image to be filtered; the embodiment of the present application does not make specific restrictions.

[0126] Take the case where the three image components of the image to be filtered are input into the filtering network at the same time and the cascade processing structure of split-merge-split is adopted as an example, see Figure 7 , which shows a schematic diagram of the structure of a loop filtering framework 70 provided in an embodiment of the present application. Figure 7As shown, the loop filtering framework 70 may include three image components of the image to be filtered (represented by Y, U, and V, respectively) 701, a first branching unit 702, first auxiliary information 703, a first processing unit 704 for the Y image component, a first processing unit 705 for the U image component, a first processing unit 706 for the V image component, second auxiliary information 707, an input fusion unit 708, a joint processing unit 709, a second branching unit 710, a second processing unit 711 for the Y image component, a second processing unit 712 for the U image component, a second processing unit 713 for the V image component, a first adder 714, a second adder 715, a third adder 716, and three image components after filtering (represented by Out_Y, Out_U, and Out_V, respectively) 717. Specifically, after the three image components 701 of the image to be filtered pass through the first branching unit 702, they will be divided into three signals: Y image component, U image component and V image component. The first Y image component and the corresponding first auxiliary information 703 enter the Y image component first processing unit 704, the second U image component and the corresponding first auxiliary information 703 enter the U image component first processing unit 705, and the third V image component and the corresponding first auxiliary information 703 enter the V image component first processing unit 706, so that three new image components will be output; the input fusion unit 708 is used to fuse the three new image components and the second auxiliary information 707, and then input them into the joint processing unit 709; the joint processing unit 709 includes a multi-layer convolution filtering network for performing convolution calculation on the input information. Since the specific convolution calculation process is similar to the relevant technical solution, the specific execution steps of the joint processing unit 709 will not be described again. After passing through the joint processing unit 709, it will enter the second branching unit 710 to be re-divided into three signals, and then the three signals are respectively input into the Y image component second processing unit 711, the U image component second processing unit 712 and the V image component second processing unit 713, and the residual information of the Y image component, the residual information of the U image component and the residual information of the V image component can be obtained in sequence; the Y image component of the three image components 701 of the image to be filtered and the obtained residual information of the Y image component are input into the first adder 714 together, and the first adder 71 4 is the filtered Y image component (represented by Out_Y); the U image component among the three image components 701 of the image to be filtered and the residual information of the obtained U image component are input into the second adder 715, and the output of the second adder 715 is the filtered U image component (represented by Out_U); the V image component among the three image components 701 of the image to be filtered and the residual information of the obtained V image component are input into the third adder 716, and the output of the third adder 716 is the filtered V image component (represented by Out_V).Here, for the output components, if only the filtered Y image component needs to be output, the loop filtering framework 70 may not include the second branch unit 710, the second adder 715 and the third adder 716; if only the filtered U image component needs to be output, the loop filtering framework 70 may not include the second branch unit 710, the first adder 714 and the third adder 716; if the filtered Y image component and the filtered U image component need to be output, the loop filtering framework 70 may not include the third adder 716; the embodiments of the present application do not make specific limitations.

[0127] Take the case where two image components of the image to be filtered are simultaneously input into the filtering network and a split-merge cascade processing structure is adopted as an example, see Figure 8 , which shows a schematic diagram of the composition structure of another loop filtering framework 80 provided in an embodiment of the present application. Figure 8 As shown, the loop filtering framework 80 may include two image components of the image to be filtered (represented by Y and U respectively) 801, a first branching unit 702, first auxiliary information 703, a first processing unit 704 for the Y image component, a first processing unit 705 for the U image component, an input fusion unit 708, a joint processing unit 709, a second processing unit 711 for the Y image component, a first adder 714 and an image component after filtering (represented by Out_Y) 802. Specifically, after the two image components 801 of the image to be filtered pass through the first branch unit 702, they will be divided into two signals: Y image component and U image component. The first Y image component and the corresponding first auxiliary information 703 enter the Y image component first processing unit 704, and the second U image component and the corresponding first auxiliary information 703 enter the U image component first processing unit 705, so that two new image components will be output; the input fusion unit 708 is used to fuse the two new image components and then input them into the joint processing unit 709; after passing through the joint processing unit 709, since only a single image component (i.e., the filtered Y image component) needs to be output, there is no need to enter the second branch unit 710 at this time, and it can be directly input into the Y image component second processing unit 711, and then the residual information of the Y image component is obtained; the Y image component in the two image components 801 of the image to be filtered and the obtained residual information of the Y image component are input into the first adder 714 together, and the output of the first adder 714 is the filtered Y image component (represented by Out_Y).

[0128] It should be noted that, since the size information of the Y image component and the U image component or the V image component may be different, Figure 7 The loop filtering framework 70 shown or Figure 8In the loop filtering framework 80 shown, an upsampling unit (or deconvolution unit or super-resolution unit) can be added before the U image component first processing unit 705 and the V image component first processing unit 706 to perform upsampling processing, so that the resolution of the upsampled U image component or the upsampled V image component is consistent with that of the Y image component, which is convenient for subsequent loop filtering processing. Figure 7 Taking the loop filtering framework 70 shown as an example, the improved loop filter in the embodiment of the present application may at least include an input fusion unit 708, a joint processing unit 709, a first adder 714, a second adder 715 and a third adder 716, but may also include a first branching unit 702, a Y image component first processing unit 704, a U image component first processing unit 705, a V image component first processing unit 706, etc., and may even include a second branching unit 710, a Y image component second processing unit 711, a U image component second processing unit 712, a V image component second processing unit 713, etc., which is not specifically limited in the embodiment of the present application.

[0129] In addition, in the embodiment of the present application, the loop filtering implementation method can adopt a split-merge-split processing structure, for example Figure 7 The loop filtering framework 70 shown in FIG. 1 may also employ a lesser split-merge processing structure, such as Figure 8 The loop filtering framework 80 shown may also adopt fewer merging-branching processing structures, or even fewer merging-branching processing structures or more branching-merging-branching-merging-branching processing structures, which is not specifically limited in the embodiments of the present application.

[0130] It should also be noted that the first auxiliary information and the second auxiliary information can all participate in the loop filtering process, for example Figure 7 The loop filtering framework 70 shown; the first auxiliary information and the second auxiliary information can also selectively participate in the loop filtering process, for example Figure 8 In the loop filtering framework 80 shown, the second auxiliary information does not participate in the loop filtering process. In the embodiment of the present application, the first auxiliary information and the second auxiliary information may both participate in the loop filtering process, or the first auxiliary information may not participate in the loop filtering process, or the second auxiliary information may not participate in the loop filtering process, or even the first auxiliary information and the second auxiliary information may not participate in the loop filtering process, and the embodiment of the present application does not specifically limit this.

[0131] It should also be noted that the fusion stages of the first auxiliary information and the second auxiliary information can be the same or different; that is, the first auxiliary information and the second auxiliary information can participate in the loop filtering process at the same stage or at different stages, which is not specifically limited in the embodiment of the present application. Figure 7 Taking the loop filtering framework 70 shown as an example, the first auxiliary information 703 and the second auxiliary information 707 can both participate in the loop filtering process in the stage corresponding to the first branch unit 702, or the first auxiliary information 703 and the second auxiliary information 707 can both participate in the loop filtering process in the stage corresponding to the input fusion unit 708, or the first auxiliary information 703 and the second auxiliary information 707 can both participate in the loop filtering process in the stage corresponding to the second branch unit 710; or the first auxiliary information 703 participates in the loop filtering process in the stage corresponding to the first branch unit 702, and the second auxiliary information 707 participates in the loop filtering process in the stage corresponding to the input fusion unit 708; or the first auxiliary information 703 participates in the loop filtering process in the stage corresponding to the first branch unit 702, and the second auxiliary information 707 participates in the loop filtering process in the stage corresponding to the input fusion unit 708; 02 participates in the loop filtering process before the stage corresponding to the first branch unit 702, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the input fusion unit 708; or, the first auxiliary information 703 participates in the loop filtering process before the stage corresponding to the first branch unit 702, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the second branch unit 710; or, the first auxiliary information 703 participates in the loop filtering process within the stage corresponding to the input fusion unit 708, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the second branch unit 710; that is, the first auxiliary information 703 and the second auxiliary information 707 can flexibly select the fusion stage in the cascade processing structure, and the embodiments of the present application do not make specific limitations.

[0132] by Figure 7Taking the loop filtering framework 70 shown as an example, it uses a deep learning network (such as CNN) for loop filtering. The difference from the traditional CNN filter is that the improved loop filter in the embodiment of the present application adopts a cascade processing structure, which can simultaneously input the three image components of the image to be filtered into the filtering network, and also incorporates other coding-related auxiliary information (such as block division information, quantization parameter information, MV information and other coding parameters), and these auxiliary information can be integrated into the filtering network at the same stage or different stages; in this way, not only the relationship between the three image components is fully utilized, but also other coding-related auxiliary information is used to assist filtering, thereby improving the filtering quality; in addition, the simultaneous processing of the three image components also effectively avoids the problem of needing to perform three complete network forward calculations on the three image components, thereby reducing the computational complexity and saving the coding bit rate. For example, taking VTM3.0 as a benchmark, it was found in an experimental test that, compared with the related technical solutions, the loop filtering implementation method of the embodiment of the present application can, under the premise of the same restored video quality, simultaneously achieve a 6.4% bit rate reduction for the Y image component, a 9.8% bit rate reduction for the U image component, and an 11.6% bit rate reduction for the V image component, thereby saving the encoding bit rate.

[0133] The above embodiment provides a loop filtering implementation method, which obtains an image to be filtered, where the image to be filtered is generated during the video encoding process from an original image in a video to be encoded, and the video to be encoded includes an original image frame, and the original image frame includes the original image; based on the image to be filtered, at least two image components of the image to be filtered are obtained respectively; fusion information of the image to be filtered is determined, where the fusion information at least includes information obtained by fusing the at least two image components; loop filtering is performed on the image to be filtered based on the fusion information to obtain at least one image component of the filtered image to be filtered; in this way, by adopting a cascade processing structure and using coding parameters such as block division information and / or QP information as auxiliary information to perform fusion processing with multiple input image components, not only is the relationship between multiple image components fully utilized, but the computational complexity is also reduced, and the encoding bit rate is saved; at the same time, the subjective and objective quality of the video reconstructed image during the encoding and decoding process is further improved.

[0134] Based on the same inventive concept as the above embodiments, see Figure 9 , which shows a schematic diagram of the structure of a loop filtering implementation device 90 provided in an embodiment of the present application. The loop filtering implementation device 90 may include: an acquisition unit 901, a branching unit 902, a determination unit 903 and a filtering unit 904, wherein,

[0135] The acquisition unit 901 is configured to acquire an image to be filtered; wherein the image to be filtered is generated during a video encoding process from an original image in a video to be encoded, the video to be encoded includes an original image frame, and the original image frame includes the original image;

[0136] The branching unit 902 is configured to obtain at least two image components of the image to be filtered based on the image to be filtered;

[0137] The determining unit 903 is configured to determine fusion information of the image to be filtered; wherein the fusion information at least includes information obtained by fusing the at least two image components;

[0138] The filtering unit 904 is configured to perform loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered after filtering.

[0139] In the above scheme, the branching unit 902 is specifically configured to perform component processing on each of the at least two original image components of the image to be filtered based on the at least two original image components of the image to be filtered, and obtain at least two image components of the image to be filtered after processing.

[0140] In the above solution, the loop filtering implementation device 90 further includes a fusion unit 905 configured to perform fusion processing on at least two image components of the image to be filtered to obtain fusion information of the image to be filtered.

[0141] In the above scheme, the determination unit 903 is further configured to determine the first auxiliary information corresponding to each original image component based on at least two original image components of the image to be filtered; wherein the first auxiliary information includes at least block division information and / or quantization parameter information.

[0142] In the above scheme, the branching unit 902 is also configured to add and process each of the at least two original image components and the first auxiliary information corresponding to each original image component based on the at least two original image components of the image to be filtered, and obtain at least two image components of the image to be filtered after processing.

[0143] In the above solution, the fusion unit 905 is further configured to perform fusion processing on at least two image components of the image to be filtered and the first auxiliary information corresponding to each original image component to obtain fusion information of the image to be filtered.

[0144] In the above solution, the demultiplexing unit 902 is further configured to respectively obtain residual information corresponding to at least one of the at least two original image components based on the fusion information;

[0145] The filtering unit 904 is specifically configured to perform a sum operation on at least one original image component of the at least two original image components and the residual information corresponding to the at least one original image component to obtain at least one image component after filtering of the image to be filtered.

[0146] In the above solution, the determining unit 903 is further configured to determine, based on at least two original image components of the image to be filtered, second auxiliary information corresponding to each original image component; wherein the second auxiliary information is different from the first auxiliary information.

[0147] In the above solution, the fusion unit 905 is further configured to perform fusion processing on at least two image components of the image to be filtered and the second auxiliary information corresponding to each original image component to obtain fusion information of the image to be filtered.

[0148] In the above scheme, the branching unit 902 is also configured to add and process each of the at least two original image components and the second auxiliary information corresponding to each original image component based on the at least two original image components of the image to be filtered, and obtain at least two image components of the image to be filtered after processing.

[0149] In the above solution, the acquisition unit 901 is specifically configured to perform video encoding processing on the original image in the video to be encoded, and use the generated reconstructed image as the image to be filtered; or,

[0150] The acquisition unit 901 is specifically configured to perform video encoding processing on the original image in the video to be encoded to generate a reconstructed image; perform preset filtering processing on the reconstructed image, and use the image after the preset filtering as the image to be filtered.

[0151] In the above scheme, see Figure 9 The loop filtering implementation device 90 further includes a sampling unit 906 configured to select a low-resolution image component for at least two original image components of the image to be filtered; and perform upsampling processing on the low-resolution image component.

[0152] It is understood that in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular system. Furthermore, the various components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The aforementioned integrated units can be implemented in the form of hardware or software functional modules.

[0153] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0154] Therefore, this embodiment provides a computer storage medium, which stores a loop filtering implementation program. When the loop filtering implementation program is executed by at least one processor, the steps of the method described in the above embodiment are implemented.

[0155] Based on the composition of the loop filtering implementation device 90 and the computer storage medium, see Figure 10 , which shows a specific hardware structure example of the loop filtering implementation device 90 provided by the embodiment of the present application, which may include: a network interface 1001, a memory 1002 and a processor 1003; each component is coupled together through a bus system 1004. It can be understood that the bus system 1004 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1004 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 10 Various buses are labeled as bus system 1004. Among them, the network interface 1001 is used to receive and send signals when sending and receiving information between other external network elements;

[0156] Memory 1002, used to store computer programs that can be run on processor 1003;

[0157] The processor 1003 is configured to, when running the computer program, execute:

[0158] Acquire an image to be filtered; wherein the image to be filtered is generated during a video encoding process from an original image in a video to be encoded, the video to be encoded includes an original image frame, and the original image frame includes the original image;

[0159] Based on the image to be filtered, respectively obtain at least two image components of the image to be filtered;

[0160] Determining fusion information of the image to be filtered; wherein the fusion information at least includes information obtained by fusing the at least two image components;

[0161] Perform loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered.

[0162] It is understood that the memory 1002 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). The memory 1002 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0163] The processor 1003 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 1003. The above-mentioned processor 1003 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented as a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1002 , and the processor 1003 reads the information in the memory 1002 and completes the steps of the above method in combination with its hardware.

[0164] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, or other electronic units or combinations thereof for performing the functions described herein.

[0165] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0166] Optionally, as another embodiment, the processor 1003 is further configured to execute the steps of the method described in the aforementioned embodiment when running the computer program.

[0167] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0168] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A loop filtering implementation method, applied to a decoding system, characterized in that: The method comprises: performing component processing on at least two image components of the image to be filtered to obtain at least two component-processed image components; performing fusion processing on at least two image components after the component processing; Performing loop filtering based on the fusion process to obtain at least one image component of the image to be filtered; Among them, the at least two image components after component processing include at least two image components among the first image component, the second image component and the third image component, and the at least one image component after filtering of the image to be filtered includes one image component among the first image component, the second image component and the third image component.

2. The method according to claim 1, characterized in that The performing the fusion processing includes inputting the at least two image components at one time.

3. The method according to claim 1, characterized in that The method further comprises: Determine first auxiliary information corresponding to at least two image components of the image to be filtered; wherein the first auxiliary information at least includes block partition information and / or quantization parameter information.

4. The method according to claim 3, characterized in that The method further comprises: Adding the at least two image components of the image to be filtered and the corresponding first auxiliary information to obtain the at least two image components of the image to be filtered.

5. The method according to claim 3, characterized in that The fusing of the at least two image components after the component processing comprises: A fusion process is performed on at least two image components of the image to be filtered and the corresponding first auxiliary information to obtain fusion information of the image to be filtered.

6. The method according to claim 1, characterized in that The performing loop filtering based on the fusion processing to obtain at least one image component after filtering of the image to be filtered includes: Based on the fusion processing, respectively obtaining residual information corresponding to at least one image component of the at least two image components of the image to be filtered; A sum operation is performed on at least one image component of the at least two image components of the image to be filtered and the corresponding residual information to obtain at least one image component of the image to be filtered after filtering.

7. The method according to claim 1, characterized in that The method further comprises: An image to be filtered is acquired, wherein the image to be filtered includes a reconstructed image, or the image to be filtered includes a reconstructed image after a preset filtering process.

8. A loop filtering implementation method, applied to a coding system, characterized in that: The method comprises: performing component processing on at least two image components of the image to be filtered to obtain at least two component-processed image components; performing fusion processing on at least two image components after the component processing; Performing loop filtering based on the fusion process to obtain at least one image component of the image to be filtered; Among them, the at least two image components after component processing include at least two image components among the first image component, the second image component and the third image component, and the at least one image component after filtering of the image to be filtered includes one image component among the first image component, the second image component and the third image component.

9. The method according to claim 8, characterized in that The performing the fusion processing includes inputting the at least two image components at one time.

10. The method according to claim 8, characterized in that The method further comprises: Determine first auxiliary information corresponding to at least two image components of the image to be filtered; wherein the first auxiliary information at least includes block partition information and / or quantization parameter information.

11. The method according to claim 10, characterized in that The method further comprises: Adding the at least two image components of the image to be filtered and the corresponding first auxiliary information to obtain the at least two image components of the image to be filtered.

12. The method according to claim 10, characterized in that The fusing of the at least two image components after the component processing comprises: A fusion process is performed on at least two image components of the image to be filtered and the corresponding first auxiliary information to obtain fusion information of the image to be filtered.

13. The method according to claim 8, characterized in that The performing loop filtering based on the fusion processing to obtain at least one image component after filtering of the image to be filtered includes: Based on the fusion processing, respectively obtaining residual information corresponding to at least one image component of the at least two image components of the image to be filtered; A sum operation is performed on at least one image component of the at least two image components of the image to be filtered and the corresponding residual information to obtain at least one image component of the image to be filtered after filtering.

14. The method according to claim 8, characterized in that The method further comprises: An image to be filtered is acquired, wherein the image to be filtered includes a reconstructed image, or the image to be filtered includes a reconstructed image after a preset filtering process.

15. A computer storage medium, characterized in that The computer storage medium stores a loop filtering implementation program. When the loop filtering implementation program is applied to a coding system, it implements the loop filtering method according to any one of claims 8 to 14 and outputs a corresponding code stream.

Images