Loop filtering implementation method and device and computer storage medium

Abstract

The embodiment of the invention discloses a loop filtering implementation method and device and a computer storage medium. The method comprises the steps of obtaining a to-be-filtered image; wherein the to-be-filtered image is generated by an original image in a to-be-coded video in a video coding process, the to-be-coded video comprises an original image frame, and the original image frame comprises the original image; respectively obtaining at least two image components of the to-be-filtered image based on the to-be-filtered image; determining fusion information of the to-be-filtered image; wherein the fusion information at least comprises fusion of the at least two image components; and performing loop filtering processing on the to-be-filtered image based on the fusion information to obtain at least one image component of the to-be-filtered image after filtering.

Description

[0001] Case Analysis This application is a divisional application of Chinese Patent No. 201980090702.4, filed on March 7, 2019, entitled "Loop Filtering Implementation Method, Apparatus and Computer Storage Medium". Technical Field

[0002] This application relates to the field of image processing technology, and in particular to a loop filtering implementation method, apparatus, and computer storage medium. Background Technology

[0003] In video encoding and decoding systems, most video coding adopts a hybrid coding framework based on block coding units (CUs). Because adjacent CUs use different coding parameters, such as different transformation processes, different quantization parameters (QP), different prediction methods, and different reference image frames, and the magnitude and distribution characteristics of the errors introduced by each CU are independent of each other, the discontinuity of the boundaries between adjacent CUs produces block artifacts, which affect the subjective and objective quality of the reconstructed image and even the prediction accuracy of subsequent encoding and decoding.

[0004] Thus, during the encoding and decoding process, loop filters are used to improve the subjective and objective quality of the reconstructed image. Traditional loop filters are usually created by manually summarizing the features of the distorted image and manually designing the filter structure and configuring the filter coefficients, such as deblocking filtering, sample adaptive compensation, and adaptive loop filtering. These manually designed filters do not fit the optimal filter well, have poor adaptability and filtering effect, and the encoder needs to write the filter-related parameters that depend on local statistical information into the bitstream to ensure consistency between the encoder and decoder, which increases the number of encoded bits.

[0005] With the rapid development of deep learning theory, the industry has proposed Convolutional Neural Networks (CNNs) to filter reconstructed images and remove image distortions, achieving significant improvements in both subjective and objective image quality compared to traditional loop filters. However, current CNN filters do not fully utilize relevant information, resulting in limited improvements in the subjective and objective image quality of the reconstructed images. Summary of the Invention

[0006] This application provides a loop filtering implementation method, apparatus, and computer storage medium. By employing a cascaded processing structure and using block partitioning information and / or QP information as auxiliary information to fuse multiple input image components, it not only fully utilizes the relationship between multiple image components but also reduces computational complexity and saves coding rate. At the same time, it further improves the subjective and objective quality of video reconstructed images during the encoding and decoding process.

[0007] The technical solution of this application embodiment can be implemented as follows: In a first aspect, embodiments of this application provide a loop filtering implementation method, the method comprising: Obtain the image to be filtered; wherein, the image to be filtered is generated during the video encoding process from the original image in the video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image; Based on the image to be filtered, at least two image components of the image to be filtered are obtained respectively; Determine the fusion information of the image to be filtered; wherein the fusion information includes at least the information obtained by fusing the at least two image components; Based on the fusion information, the image to be filtered is subjected to loop filtering to obtain at least one image component of the filtered image.

[0008] Secondly, embodiments of this application provide a loop filtering implementation device, which includes: an acquisition unit, a splitting unit, a determination unit, and a filtering unit, wherein... The acquisition unit is configured to acquire an image to be filtered; wherein the image to be filtered is generated during the video encoding process from the original image in the video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image; The splitting unit is configured to obtain at least two image components of the image to be filtered based on the image to be filtered. The determining unit is configured to determine the fusion information of the image to be filtered; wherein the fusion information includes at least the information obtained by fusing the at least two image components; The filtering unit is configured to perform loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered.

[0009] Thirdly, embodiments of this application provide a loop filter implementation device, the loop filter implementation device comprising: a memory and a processor, wherein, The memory is used to store computer programs that can run on the processor; The processor is configured to perform the steps of the method as described in the first aspect when running the computer program.

[0010] Fourthly, embodiments of this application provide a computer storage medium storing a loop filter implementation program, which, when executed by at least one processor, implements the steps of the method described in the first aspect.

[0011] This application provides a loop filtering implementation method, apparatus, and computer storage medium. First, an image to be filtered is acquired, which is generated during the video encoding process from the original image in the video to be encoded. Then, based on the image to be filtered, at least two image components are obtained; this process can be considered a decoupling stage. Next, fusion information of the image to be filtered is determined; wherein, the fusion information includes at least the fusion of the at least two image components; this process can be considered a merging stage. Thus, by employing a cascaded processing structure and fusing multiple input image components, not only is the relationship between the multiple image components fully utilized, but the problem of needing to perform multiple complete network forward calculations on these multiple image components is also effectively avoided, thereby reducing computational complexity and saving encoding bitrate. Finally, based on the fusion information, loop filtering is performed on the image to be filtered to obtain at least one image component of the filtered image. Thus, the fusion information can further assist filtering, improving the subjective and objective quality of the video reconstructed image during the encoding and decoding process. Attached Figure Description

[0012] Figure 1 A schematic diagram illustrating the composition structure of a traditional coding block diagram for related technical solutions; Figure 2 This is a schematic diagram illustrating the composition structure of an improved coding block diagram provided in an embodiment of this application; Figure 3 A flowchart illustrating a loop filtering implementation method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a block partitioning matrix provided in an embodiment of this application; Figure 5 A schematic diagram of the composition structure of a conventional CNN filter provided in an embodiment of this application; Figure 6A and Figure 6B A schematic diagram illustrating the composition structure of another conventional CNN filter provided in an embodiment of this application; Figure 7 A schematic diagram of the composition structure of a loop filter framework provided in an embodiment of this application; Figure 8A schematic diagram of the composition structure of another loop filter framework provided in an embodiment of this application; Figure 9 This is a schematic diagram of the composition structure of a loop filtering implementation device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the specific hardware structure of a loop filtering implementation device provided in an embodiment of this application. Detailed Implementation

[0013] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0014] In video encoding and decoding systems, the video to be encoded includes original image frames, which in turn contain original images. These original images undergo various processing steps, such as prediction, transformation, quantization, reconstruction, and filtering. During these processes, the processed video image may have undergone pixel value shifts relative to the original image, leading to visual impairments or artifacts. Furthermore, in the hybrid coding framework based on block-shaped CUs used in most video encoding and decoding systems, adjacent coding blocks employ different coding parameters (e.g., different transformation processes, different QPs, different prediction methods, different reference image frames, etc.). The magnitude and distribution characteristics of the errors introduced by each coding block are independent, and the discontinuity at the boundaries of adjacent coding blocks produces block artifacts. These distortions not only affect the subjective and objective quality of the reconstructed image, but if the reconstructed image is used as a reference image for subsequent encoded pixels, it can even affect the prediction accuracy of subsequent encoding and decoding, thus affecting the bit size in the video bitstream. Therefore, in-loop filters are often added to video encoding and decoding systems to improve the subjective and objective quality of the reconstructed image.

[0015] See Figure 1 It illustrates a traditional coding frame provided by the relevant technical solution. Figure 10 A schematic diagram of its composition and structure. (See attached diagram.) Figure 1 As shown, this traditional coding frame Figure 10The system may include components such as a transform and quantization unit 101, an inverse transform and inverse quantization unit 102, a prediction unit 103, a filtering unit 104, and an entropy coding unit 105; wherein, the prediction unit 103 further includes an intra-frame prediction unit 1031 and an inter-frame prediction unit 1032. For the input raw image, a coding tree unit (CTU) can be obtained through preliminary partitioning, and a CTU can be further partitioned by content adaptive partitioning to obtain a CU, which generally contains one or more coding blocks (CBs). By performing intra-frame prediction by the intra-frame prediction unit 1031 or inter-frame prediction by the inter-frame prediction unit 1032 on the coded block, residual information can be obtained. This residual information is then transformed by the transform and quantization unit 101, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate. After determining the prediction mode, the prediction unit 103 also provides the selected intra-frame prediction data or inter-frame prediction data to the entropy coding unit 105. Furthermore, the inverse transform and inverse quantization unit 102 is used to reconstruct the coded block, reconstructing the residual block in the pixel domain. This reconstructed residual block is then filtered by the filtering unit 104 to remove block artifacts, and then added to the decoded image buffer unit to generate a reconstructed reference image. The entropy coding unit 105 is used to encode various coding parameters and quantized transform coefficients. For example, the entropy coding unit 105 uses header information encoding and context-based adaptive binary arithmetic coding. The Coding (CABAC) algorithm can be used to encode the coding information of the determined prediction pattern and output the corresponding bitstream.

[0016] against Figure 1 The traditional coding frame Figure 10The filtering unit 104 is a loop filter, also known as an in-loop filter. It can include a de-blocking filter (DBF) 1041, a sample adaptive offset (SAO) filter 1042, and an adaptive loop filter (ALF) 1043. The de-blocking filter 1041 is used to implement de-blocking filtering. In the next-generation video coding standard H.266 / Versatile Video Coding (VVC), for all coded block boundaries in the original image, the boundary strength is first determined based on the coding parameters on both sides of the boundary, and the calculated block boundary texture value is used to determine whether to perform de-blocking filtering. Finally, the pixel information on both sides of the coded block boundary is corrected based on the boundary strength and the filtering decision. In VVC, after deblocking filtering is performed, SAO (Sample Adaptive Compensation Filter 1042) is introduced to reduce quantization distortion of high-frequency AC coefficients. Furthermore, starting from the pixel domain, negative values ​​are added to pixels at peaks and positive values ​​to pixels at troughs for compensation. In VVC, after deblocking filtering and sample adaptive compensation filtering, further filtering is performed using an adaptive loop filter 1043. For adaptive loop filtering, the optimal filter in the mean square sense is calculated based on the pixel values ​​of the original image and the distorted image. However, these filters (such as the deblocking filter 1041, the sample adaptive compensation filter 1042, and the adaptive loop filter 1043) not only require meticulous manual design and a large number of judgments and decisions, but also require the encoding end to write filter-related parameters (such as filter coefficients and flag values ​​indicating whether to use the filter) that depend on local statistical information into the bit stream to ensure consistency between the encoding and decoding ends, which increases the number of encoded bits. At the same time, the manually designed filters do not fit the complex function of the real optimization target well, and the filtering effect needs to be improved.

[0017] This application provides a loop filtering implementation method, which is applied to an improved coding block diagram; and Figure 1 The traditional coding frame shown Figure 10The main difference is that an improved loop filter is used to replace the deblocking filter 1041, sample adaptive compensation filter 1042, and adaptive loop filter 1043 in the related technical solutions. In the embodiments of this application, the improved loop filter can be a Convolutional Neural Network (CNN) filter or a filter established by other deep learning methods; the embodiments of this application do not impose specific limitations.

[0018] For example, see convolutional neural network filters. Figure 2 This illustrates a schematic diagram of the composition structure of an improved coding block diagram 20 provided in an embodiment of this application. For example... Figure 2 As shown, compared with traditional coding boxes Figure 10 In contrast, the filtering unit 104 in the improved coding block diagram 20 includes a convolutional neural network filter 201. This convolutional neural network filter 201 can completely replace... Figure 1 The deblocking filter 1041, sample adaptive compensation filter 1042, and adaptive loop filter 1043 in the filter can also partially replace the other filters. Figure 1 Any one or two of the deblocking filter 1041, the sample adaptive compensation filter 1042, and the adaptive loop filter 1043, or even together with... Figure 1 The deblocking filter 1041, the sample adaptive compensation filter 1042, and the adaptive loop filter 1043 can be used in combination, or any one or more of them. It should also be noted that, for... Figure 1 or Figure 2 Each component shown, such as the transform and quantization unit 101, the inverse transform and inverse quantization unit 102, the prediction unit 103, the filtering unit 104, the entropy coding unit 105, or the convolutional neural network filter 201, can be a virtual module or a hardware module. Furthermore, those skilled in the art will understand that these units do not constitute a limitation on the coding block diagram, which may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0019] In this embodiment, the convolutional neural network filter 201, after being trained, can be directly deployed at the encoding and decoding ends, thus eliminating the need to transmit any filter-related parameters. Furthermore, the convolutional neural network filter 201 can fuse auxiliary information such as block partitioning information and / or QP information with multiple input image components. This not only fully utilizes the relationships between multiple image components but also reduces computational complexity and saves encoding bitrate. At the same time, it further improves the subjective and objective quality of the video reconstructed image during the encoding and decoding process.

[0020] It should be noted that the loop filtering implementation method of this application embodiment can be applied not only to encoding systems but also to decoding systems. Generally, in order to save encoding bitrate while ensuring that the decoding system can perform correct decoding processing, the loop filter of this application embodiment must be deployed synchronously in both the encoding and decoding systems. The following will describe its application in an encoding system as an example in detail.

[0021] See Figure 3 The diagram illustrates a flowchart of a loop filtering implementation method provided in an embodiment of this application. The method may include: S301: Obtain the image to be filtered; wherein, the image to be filtered is generated during the video encoding process from the original image in the video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image; S302: Based on the image to be filtered, obtain at least two image components of the image to be filtered; S303: Determine the fusion information of the image to be filtered; wherein the fusion information includes at least the information obtained by fusing the at least two image components; S304: Perform loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the filtered image.

[0022] It should be noted that since the original image can be divided into CTUs or CTUs into CUs, the block partitioning information in this application embodiment can refer to CTU partitioning information or CU partitioning information. Thus, the loop filtering implementation method in this application embodiment can be applied not only to loop filtering at the CU level but also to loop filtering at the CTU level. This application embodiment does not make any specific limitations.

[0023] In this embodiment, after acquiring the image to be filtered, at least two image components of the image to be filtered are obtained based on the image to be filtered. This process can be regarded as a splitting stage, used to obtain at least two image components respectively. Then, the fusion information of the image to be filtered is determined. The fusion information includes at least the information obtained by fusing the at least two image components. This process can be regarded as a merging stage, used to fuse the at least two image components. Thus, this embodiment adopts a cascaded processing structure. By fusing multiple input image components, it not only makes full use of the relationship between multiple image components, but also effectively avoids the problem of having to perform multiple complete forward network calculations on these multiple image components, thereby reducing computational complexity and saving coding rate. Finally, based on the fusion information, the image to be filtered is subjected to loop filtering to obtain at least one image component of the filtered image. In this way, the fusion information can further assist filtering, improving the subjective and objective quality of the video reconstructed image during the encoding and decoding process.

[0024] In some embodiments, the image components include a first image component, a second image component, and a third image component; wherein the first image component represents a luminance component, the second image component represents a first chrominance component, and the third image component represents a second chrominance component.

[0025] It should be noted that in video images, a first image component, a second image component, and a third image component are generally used to represent the original image or the image to be filtered. In the luminance-chrominance component representation method, these three image components are a luminance component, a blue chrominance (color difference) component, and a red chrominance (color difference) component, respectively. Specifically, the luminance component is usually represented by the symbol Y, the blue chrominance component is usually represented by the symbol Cb (or U), and the red chrominance component is usually represented by the symbol Cr (or V). In the embodiments of this application, the first image component can be the luminance component Y, the second image component can be the blue chrominance component U, and the third image component can be the red chrominance component V; however, the embodiments of this application do not impose specific limitations. At least one image component represents one or more of the first, second, and third image components, while at least two image components can be the first, second, and third image components, or they can be the first and second image components, or the first and third image components, or even the second and third image components; the embodiments of this application do not impose specific limitations in any of these cases.

[0026] In the next-generation video coding standard VVC, the corresponding test model is the VVC TestModel (VTM). When implementing VTM testing, the current standard test sequence uses a 4:2:0 YUV format. In this format, each frame of the video to be encoded consists of three image components: one luminance component (represented by Y) and two chrominance components (represented by U and V). Assuming the original image in the video to be encoded has a height of H and a width of W, the size information corresponding to the first image component is... The size information corresponding to the second or third image component is... It should be noted that this application embodiment will be described using YUV in 4:2:0 format as an example, but the loop filtering implementation method of this application embodiment is also applicable to other sampling formats.

[0027] Taking YUV in 4:2:0 format as an example, since the size information of the first image component is different from that of the second or third image component, in order to input the first image component and / or the second image component and / or the third image component into the loop filter model at the same time, it is necessary to sample or reassemble these three image components so that the spatial size information of the three image components is the same.

[0028] In some embodiments, pixel rearrangement (also known as downsampling) can be performed on the high-resolution image components to ensure that the spatial size information of the three image components is the same. Specifically, before obtaining at least two image components of the image to be filtered based on the image to be filtered, the method further includes: For at least two original image components of the image to be filtered, select the high-resolution image component; The high-resolution image components are subjected to pixel rearrangement processing.

[0029] It should be noted that the three image components (e.g., the first, second, and third image components) contained in the original image are the original image components before any further processing. If the first image component is the luminance component, the second image component is the first chrominance component, and the third image component is the second chrominance component, then the high-resolution image component is the first image component, and pixel rearrangement processing is required for the first image component. For example, taking a 2×2 original image as an example, converting it to 4 channels, that is, arranging a 2×2×1 tensor into a 1×1×4 tensor; then when the size information of the first image component of the original image is... At that time, it can be converted to pixel rearrangement before performing loop filtering. The form; since the size information of the second and third image components are both This ensures that the spatial dimensions of the three image components are identical; subsequently, the first, second, and third image components, after pixel rearrangement processing, are merged to transform the image into... An improved loop filter with input in the form of a .

[0030] In some embodiments, the low-resolution image components may also be upsampled to ensure that the spatial size information of the three image components is the same. Specifically, before obtaining at least two image components of the image to be filtered based on the image to be filtered, the method further includes: For at least two original image components of the image to be filtered, select the low-resolution image component; The low-resolution image components are upsampled.

[0031] It should be noted that, in addition to pixel rearrangement processing (i.e., downward adjustment) of size information for high-resolution image components, this embodiment can also perform upsampling processing (i.e., upward adjustment) on low-resolution image components. Furthermore, for low-resolution image components, not only upsampling processing but also deconvolution processing and even super-resolution processing can be performed. These three processing methods have the same effect, and this embodiment does not impose specific limitations on them.

[0032] It should also be noted that the three image components (e.g., the first, second, and third image components) contained in the original image are the original image components before any further processing. If the first image component is the luminance component, the second image component is the first chrominance component, and the third image component is the second chrominance component, then the low-resolution image component is either the second or third image component. In this case, upsampling processing needs to be performed on either the second or third image component. For example, when the size information of the second and third image components of the original image is... At that time, it can be converted to [a specific value] through upsampling before performing loop filtering. The form; because the size information of the first image component is This also ensures that the spatial size information of the three image components is the same, and the resolution of the upsampled second and third image components will be consistent with that of the first image component.

[0033] In some embodiments, acquiring the image to be filtered includes: The original image in the video to be encoded is subjected to video encoding processing, and the resulting reconstructed image is used as the image to be filtered; or... The original image in the video to be encoded is processed by video encoding to generate a reconstructed image; the reconstructed image is then subjected to a preset filtering process, and the image after preset filtering is used as the image to be filtered.

[0034] It should be noted that, based on the improved coding block diagram 20, during the video coding process of the original image in the video to be coded, the original image undergoes CU partitioning, prediction, transformation, and quantization. Furthermore, to obtain a reference image for video coding of subsequent images to be coded, inverse transformation and inverse quantization, reconstruction, and filtering can also be performed. Thus, the image to be filtered in this embodiment can be a reconstructed image generated after reconstruction during video coding, or a pre-filtered image obtained by applying other pre-defined filtering methods (such as deblocking filtering). This embodiment does not impose specific limitations.

[0035] In some embodiments, before obtaining at least two image components of the image to be filtered, the method further includes: Based on at least two original image components of the image to be filtered, first auxiliary information corresponding to each original image component is determined; wherein, the first auxiliary information includes at least block partitioning information and / or quantization parameter information.

[0036] Understandably, the first auxiliary information can be used to assist filtering and improve filtering quality. In the embodiments of this application, the first auxiliary information can be not only block partitioning information (such as CU partitioning information and / or CTU partitioning information), but also quantization parameter information, and even motion vector (MV) information, prediction direction information, etc.; these information can be used as the first auxiliary information alone, or can be arbitrarily combined as the first auxiliary information, such as using block partitioning information alone as the first auxiliary information, or using block partitioning information and quantization parameter information together as the first auxiliary information, or using block partitioning information and MV information together as the first auxiliary information, etc. The embodiments of this application do not make specific limitations.

[0037] Optionally, in some embodiments, obtaining at least two image components of the image to be filtered based on the image to be filtered includes: Based on at least two original image components of the image to be filtered, component processing is performed on each of the at least two original image components to obtain at least two image components of the image to be filtered.

[0038] Optionally, in some embodiments, obtaining at least two image components of the image to be filtered based on the image to be filtered includes: Based on at least two original image components of the image to be filtered, each original image component and its corresponding first auxiliary information are added to each of the at least two original image components to obtain at least two image components of the image to be filtered.

[0039] It should be noted that "obtaining at least two image components of the image to be filtered" can be regarded as the first splitting stage. In this way, for the at least two original image components of the image to be filtered, component processing (such as deep learning) can be performed on them respectively, so as to obtain at least two image components; in addition, the first auxiliary information corresponding to each original image component can be added to the corresponding image component, so as to obtain at least two image components; that is, for the first splitting stage, the first auxiliary information can be added or not, and the embodiments of this application do not make specific limitations.

[0040] Optionally, in some embodiments, determining the fusion information of the image to be filtered includes: At least two image components of the image to be filtered are fused to obtain the fusion information of the image to be filtered.

[0041] Optionally, in some embodiments, determining the fusion information of the image to be filtered includes: The image to be filtered is fused together with at least two image components and the first auxiliary information corresponding to each original image component to obtain the fused information of the image to be filtered.

[0042] It should be noted that "determining the fusion information of the image to be filtered" can be regarded as the merging stage. Thus, the fusion information of the image to be filtered can be obtained by directly fusing at least two image components, or it can be obtained by fusing at least two image components and their corresponding first auxiliary information. That is to say, for the merging stage, the first auxiliary information can be added or not, and this application embodiment does not make specific limitations.

[0043] In relevant technical solutions, such as Figure 1 The filter in the filter unit 104 shown processes the three image components independently, requiring three complete network forward calculations. However, the loop filtering implementation method of this application embodiment can input the three image components at once by fusing information, or input any two of the three image components at the same time. This application embodiment does not make specific limitations. This effectively avoids the problem of needing to perform multiple complete network forward calculations for multiple image components and reduces computational complexity.

[0044] It can also be understood that since the original image can be divided into CTUs or CTUs into CUs, the loop filtering implementation method of this application embodiment can be applied not only to loop filtering at the CU level (where the block partitioning information is CU partitioning information) but also to loop filtering at the CTU level (where the block partitioning information is CTU partitioning information). This application embodiment does not impose specific limitations. The following description will use CU partitioning information as block partitioning information as an example.

[0045] In some embodiments, determining the first auxiliary information corresponding to each image component includes: Based on the original image in the video to be encoded, each image component in at least two image components of the original image is divided into a coding unit (CU) to obtain CU division information. The CU division information is then used as the block division information corresponding to each image component of the image to be filtered.

[0046] Further, in some embodiments, using the CU partitioning information as the block partitioning information corresponding to each image component of the image to be filtered includes: For the CU partitioning information, a first value is filled at each pixel position corresponding to the CU boundary, and a second value is filled at other pixel positions to obtain a first matrix corresponding to the CU partitioning information; wherein the first value and the second value are different. The first matrix is ​​used as the block partitioning information corresponding to each image component of the image to be filtered.

[0047] It should be noted that the first value can be a preset numerical value, letter, etc., and the second value can also be a preset numerical value, letter, etc. The first value and the second value are different; for example, the first value can be set to 2 and the second value can be set to 1, but the embodiments of this application do not make specific limitations.

[0048] In this embodiment, CU partitioning information can be used as first auxiliary information to assist in filtering the image to be filtered. That is, during the video encoding process of the original image in the video to be encoded, the CU partitioning information can be fully utilized to fuse it with at least two image components of the image to be filtered to guide the filtering process.

[0049] Specifically, the CU partitioning information is converted into a Coding Unit Map (CUmap) and represented as a two-dimensional matrix, namely the CUmap matrix, which is also the first matrix in this embodiment. That is, taking the first image component of the original image as an example, it can be divided into multiple CUs. The pixel positions corresponding to the boundaries of each CU are filled with a first value, while the other pixel positions are filled with a second value, thus constructing a first matrix reflecting the CU partitioning information. For example, see [link to example]. Figure 4 This illustrates a schematic diagram of a block partitioning matrix provided in an embodiment of this application. Figure 4 As shown, if the image represents a CTU, then the CTU can be divided into 9 CUs. Assuming the first value is set to 2 and the second value is set to 1, then each pixel position corresponding to the boundary of each CU is filled with 2, while other pixel positions are filled with 1. That is to say, the pixel positions filled with 2 represent the boundaries of the CU, thereby determining the CU division information, that is, the first auxiliary information corresponding to the first image component of the image to be filtered.

[0050] It should also be noted that if the first image component is a luminance component, and the second and third image components are both chrominance components, then the CU partitioning information of the first image component may differ from that of the second or third image component. Therefore, when the CU partitioning information of the first image component differs from that of the second or third image component, it is necessary to determine the CU partitioning information corresponding to the first image component and the second or third image component of the image to be filtered, respectively; then, this information is added as first auxiliary information to the corresponding first, second, or third image component. When the CU partitioning information of the first image component is the same as that of the second or third image component, only the CU partitioning information of the first, second, or third image component needs to be determined, and then the determined CU partitioning information is added as first auxiliary information to the corresponding first, second, or third image component. This facilitates the subsequent fusion of at least two new image components for loop filtering of the image to be filtered.

[0051] In some embodiments, determining the first auxiliary information corresponding to each image component includes: Based on the original image in the video to be encoded, the quantization parameters corresponding to each of the at least two image components of the original image are obtained respectively, and the quantization parameters are used as the quantization parameter information corresponding to each image component of the image to be filtered.

[0052] Further, in some embodiments, the step of using the quantization parameters as quantization parameter information corresponding to each image component of the image to be filtered includes: A second matrix with the same size as each image component of the original image is established; wherein, each pixel position in the second matrix is ​​filled with the normalized value of the quantization parameter corresponding to each image component of the original image; The second matrix is ​​used as the quantization parameter information corresponding to each image component of the image to be filtered.

[0053] It should be noted that different quantization parameters result in varying degrees of distortion in the images being filtered. Incorporating quantization parameter information allows the filtering network to adaptively process arbitrary quantization parameters during training.

[0054] In this embodiment, quantization parameter information can also be used as first auxiliary information to assist in filtering the image to be filtered. That is, during the video encoding process of the original image in the video to be encoded, quantization parameter information can be fully utilized to fuse it with at least two image components of the image to be filtered, thereby guiding the filtering process. The quantization parameter information can be normalized or non-normalized (e.g., classification processing, interval partitioning processing, etc.); the normalization processing of quantization parameters will be described in detail below as an example.

[0055] Specifically, the quantization parameter information is converted into a second matrix reflecting the quantization parameter information; that is, taking the first image component of the original image as an example, a matrix with the same size as the first image component of the original image is established, and each pixel position in this matrix is ​​filled with the normalized value of the quantization parameter corresponding to the first image component of the original image; wherein, the normalized value of the quantization parameter is... This means, that is: (1) In equation (1), This represents the quantization parameter value corresponding to the first image component of the original image. This represents the x-coordinate value of the position of each pixel in the first image component of the original image. The ordinate value represents the position of each pixel in the first image component of the original image; This represents the maximum value of the quantization parameter. Generally speaking, The value is 51, but Other values ​​are also possible, such as 29, 31, etc., but this application does not impose specific limitations on the embodiments.

[0056] In some embodiments, before obtaining at least two image components of the image to be filtered, the method further includes: Based on at least two original image components of the image to be filtered, second auxiliary information corresponding to each original image component is determined; wherein the second auxiliary information is different from the first auxiliary information.

[0057] Optionally, in some embodiments, determining the fusion information of the image to be filtered includes: The image to be filtered is fused together with at least two image components and the second auxiliary information corresponding to each original image component to obtain the fused information of the image to be filtered.

[0058] Optionally, in some embodiments, obtaining at least two image components of the image to be filtered based on the image to be filtered includes: Based on at least two original image components of the image to be filtered, each original image component and its corresponding second auxiliary information are added to each of the at least two original image components to obtain at least two image components of the image to be filtered.

[0059] It should be noted that the first auxiliary information and the second auxiliary information are different. Both the first and second auxiliary information can be used to assist filtering and improve the filtering quality. In the embodiments of this application, the first auxiliary information can be one or more of block partitioning information, quantization parameter information, MV information, and prediction direction information, and the second auxiliary information can also be one or more of block partitioning information, quantization parameter information, MV information, and prediction direction information; that is, when the first auxiliary information is block partitioning information, the second auxiliary information can be quantization parameter information; or, when the first auxiliary information is quantization parameter information, the second auxiliary information can be block partitioning information; or, when the first auxiliary information is block partitioning information and quantization parameter information, the second auxiliary information can be MV information; or, when the first auxiliary information is block partitioning information, the second auxiliary information can be quantization parameter information and MV information; the embodiments of this application do not make specific limitations.

[0060] It should also be noted that the fusion stages of the first auxiliary information and the second auxiliary information can be the same or different. In the embodiments of this application, it is assumed that the first splitting stage is used to represent the processing stage corresponding to obtaining at least two image components of the image to be filtered, the merging stage is used to represent the processing stage corresponding to determining the fusion information of the image to be filtered, and the second splitting stage is used to represent the processing stage corresponding to determining the residual information of each image component after the fusion process. Thus, the fusion stage of the first auxiliary information can be any one of the first splitting stage, the merging stage, or the second splitting stage, and the fusion stage of the second auxiliary information can also be any one of the first splitting stage, the merging stage, or the second splitting stage; that is, the fusion stage of the first auxiliary information can be the first splitting stage, and the fusion stage of the second auxiliary information can be the merging stage; or, the fusion stage of the first auxiliary information can be the merging stage, and the fusion stage of the second auxiliary information can be the first splitting stage; or, the fusion stage of the first auxiliary information can be the second splitting stage, and the fusion stage of the second auxiliary information can be the merging stage; or, the fusion stage of the first auxiliary information can be the first splitting stage, and the fusion stage of the second auxiliary information can also be the first splitting stage; or, the fusion stage of the first auxiliary information can be the merging stage, and the fusion stage of the second auxiliary information can also be the merging stage; the embodiments of this application do not impose specific limitations.

[0061] In some embodiments, performing loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the filtered image includes: Based on the fusion information, residual information corresponding to at least one of the at least two original image components is obtained respectively; The residual information corresponding to at least one of the at least two original image components is summed with the residual information corresponding to the at least one original image component to obtain at least one image component of the image to be filtered.

[0062] It should be noted that the loop filtering implementation method in this application adopts a multi-stage cascaded processing structure, such as a split-merge-split processing structure, a split-merge processing structure, or a merge-split processing structure, etc. This application does not make specific limitations.

[0063] Specifically, if we first need to obtain at least two image components of the image to be filtered (i.e., the first splitting stage), and then fuse these two image components (i.e., the merging stage), then after all information fusion processing, when multiple image components need to be output simultaneously, such as the first image component, the second image component, and the third image component, we can obtain the residual information corresponding to the first image component, the second image component, and the third image component by jointly processing the fused information. Then, we can sum the residual information of the first image component with the residual information of the first image component, sum the residual information of the second image component with the residual information of the second image component, and sum the residual information of the third image component with the residual information of the third image component, respectively, to obtain the filtered first image component, the filtered second image component, and the filtered third image component of the image to be filtered. This process is the second splitting stage. Therefore, the entire loop filtering process adopts a splitting-merging-splitting processing structure. If we first need to obtain at least two image components of the image to be filtered (i.e., the first splitting stage), and then fuse these two image components (i.e., the merging stage), then after all the information fusion processing, when only one image component needs to be output, such as the first image component, we can obtain the residual information corresponding to the first image component by performing joint processing on the fused information. Then, we can sum the first image component and its corresponding residual information to obtain the filtered first image component of the image to be filtered. This process does not have a second splitting stage. Therefore, the entire loop filtering process adopts a splitting-merging processing structure. In addition, if it is not necessary to obtain at least two image components of the image to be filtered separately, that is, if the first splitting stage is not required, the at least two image components of the image to be filtered can be directly fused, that is, directly enter the merging stage; after all information fusion processing, since multiple image components need to be output at the same time, a second splitting stage is also required; so the entire loop filtering process adopts a merging-splitting processing structure.

[0064] It should also be noted that the loop filtering implementation method of this application embodiment can also employ more cascaded processing structures, such as a split-merge-split-merge-split processing structure, etc. For these cascaded processing structures, this application embodiment can employ a typical cascaded structure, such as a split-merge-split processing structure, or fewer cascaded processing structures than the typical cascaded structure, such as a split-merge processing structure or a merge-split processing structure, etc.; it can even employ more cascaded processing structures than the typical cascaded structure, such as a split-merge-split-merge-split processing structure, etc., and this application embodiment does not impose specific limitations.

[0065] In some embodiments, the improved loop filter includes a convolutional neural network filter.

[0066] It should be noted that the improved loop filter is used to implement loop filtering processing on the image to be filtered. The improved loop filter can be a convolutional neural network filter or a filter established using other deep learning methods; this application does not specifically limit the specific type. Here, a convolutional neural network filter, also known as a CNN filter, is a type of feedforward neural network that includes convolutional computation and has a deep structure; it is one of the representative algorithms of deep learning. The input layer of a CNN filter can process multidimensional data, such as the three image components (Y / U / V) channels of the original image in the video to be encoded.

[0067] See Figure 5 This illustrates a schematic diagram of the composition structure of a conventional CNN filter 50 provided in an embodiment of this application. Figure 5 As shown, this traditional CNN filter 50 is an improvement on the previous generation video coding standard H.265 / High Efficiency Video Coding (HEVC). It contains a two-layer convolutional network structure and can replace the deblocking filter and the sample adaptive compensation filter. The image to be filtered (using...) (This indicates that) the input is fed into the input layer of a traditional CNN filter 50, and then sequentially passes through the first convolutional network layer. (Assuming the convolutional kernel size is 3×3, containing 64 feature maps) and the second convolutional network (Assuming the convolution kernel size is 5×5 and contains 32 feature maps) then, a residual information is obtained. Then the image to be filtered and residual information The summation operation is performed to obtain the filtered image output by the traditional CNN filter 50 (using...). (Represented). This convolutional network structure, also known as a residual neural network, is used to output the residual information corresponding to the image to be filtered. In this traditional CNN filter 50, the three image components (Y / U / V) of the image to be filtered are processed independently, but they share the same filtering network and its related parameters.

[0068] See Figure 6A and Figure 6B It illustrates a schematic diagram of the composition structure of another conventional CNN filter 60 provided in an embodiment of this application; this conventional CNN filter 60 uses two filtering networks, such as Figure 6A The filter network shown is dedicated to outputting the first image component, such as Figure 6BThe filter network shown is dedicated to outputting either the second or third image component. Assuming the original image in the video to be encoded has a height of H and a width of W, then the size information corresponding to the first image component is: The first image component can be pixel rearranged to convert it into... The form; since the size information corresponding to the second or third image component is... Therefore, merging these three image components transforms them into... The input is in the form of a traditional CNN filter 60. Based on, as shown in... Figure 6A The filtering network shown receives the image to be filtered in its input layer. (Assuming the kernel size is N×N and the number of channels is 6) After that, it passes through the first convolutional network layer. (Assuming the kernel size is L1×L1, the number of kernels is M, and the number of channels is 6) and the second convolutional network (Assuming the kernel size is L2×L2, the number of kernels is 4, and the number of channels is M) then, a residual information is obtained. (Assuming the kernel size is N×N and the number of channels is 4); then the input image to be filtered is... and residual information The summation operation is performed to obtain the first image component (using the traditional CNN filter 60) after filtering. (Indicated). Based on such Figure 6B The filtering network shown receives the image to be filtered in its input layer. (Assuming the kernel size is N×N and the number of channels is 6) After that, it passes through the first convolutional network layer. (Assuming the kernel size is L1×L1, the number of kernels is M, and the number of channels is 6) and the second convolutional network (Assuming the kernel size is L2×L2, the number of kernels is 2, and the number of channels is M) then, a residual information is obtained. (Assuming the kernel size is N×N and the number of channels is 2); then the input image to be filtered is... and residual information The summation operation is performed to finally obtain the filtered second image component or the filtered third image component output by the traditional CNN filter 60 (using...). express).

[0069] against Figure 5 The traditional CNN filter 50 shown, or Figure 6A and Figure 6BThe traditional CNN filter 60 shown is unreasonable because it does not consider the relationship between different image components and processes each image component independently. Furthermore, it does not fully utilize block partitioning information, QP information, and other encoding parameters at the input. However, the distortion of the reconstructed image mainly comes from block artifacts, and the boundary information of block artifacts is determined by CU partitioning information. In other words, the filtering network in the CNN filter should focus on boundary regions. In addition, incorporating quantization parameter information into the filtering network helps improve its generalization ability, enabling it to filter distorted images of arbitrary quality. Therefore, the loop filtering implementation method provided in this application not only has a reasonable CNN filtering structure, allowing the same filtering network to simultaneously receive multiple image components, but also fully considers the relationship between these multiple image components, and can simultaneously output enhanced images of these image components after filtering. Furthermore, this loop filtering implementation method can also improve the filtering quality by incorporating block partitioning information and / or QP information and other encoding parameters as auxiliary information for auxiliary filtering.

[0070] It should be noted that, in the embodiments of this application, "obtaining at least two image components of the image to be filtered" can specifically mean determining auxiliary information (such as first auxiliary information or second auxiliary information) for each of the first, second, and third image components of the image to be filtered, and obtaining three image components after adding the information; it can also mean determining auxiliary information for each of the first and second image components of the image to be filtered, and obtaining two image components after adding the information; it can also mean determining auxiliary information for each of the first and third image components of the image to be filtered, and obtaining two image components after adding the information; or it can even mean determining auxiliary information for each of the second and third image components of the image to be filtered, and obtaining two new image components after adding the information. The embodiments of this application do not impose specific limitations on this.

[0071] It should also be noted that the fusion information of the image to be filtered can be obtained by directly fusing at least two image components, or it can be obtained by fusing at least two image components and corresponding auxiliary information (such as first auxiliary information or second auxiliary information); the embodiments of this application do not make specific limitations.

[0072] If the fusion information is obtained by directly fusing at least two image components, then the first, second, and third image components of the image to be filtered can be fused to obtain the fusion information; the first and second image components of the image to be filtered can be fused to obtain the fusion information; the first and third image components of the image to be filtered can be fused to obtain the fusion information; or even the second and third image components of the image to be filtered can be fused to obtain the fusion information.

[0073] If the fusion information is obtained by fusing at least two image components and corresponding auxiliary information (such as first or second auxiliary information), then it can be obtained by fusing the first, second, and third image components of the image to be filtered with the auxiliary information; it can also be obtained by fusing the first and second image components of the image to be filtered with the auxiliary information; it can also be obtained by fusing the first and third image components of the image to be filtered with the auxiliary information; or it can even be obtained by fusing the second and third image components of the image to be filtered with the auxiliary information. Specifically, for "obtained by fusing at least two image components and corresponding auxiliary information (such as first or second auxiliary information)," it can be obtained by first fusing at least two image components of the image to be filtered and then incorporating the auxiliary information; it can also be obtained by first incorporating each of the at least two image components of the image to be filtered with the corresponding auxiliary information, and then fusing the processed at least two image components; that is, the specific method of fusion processing is not specifically limited in the embodiments of this application.

[0074] Furthermore, in the embodiments of this application, "performing loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered" specifically means that after multiple image components (such as a first image component, a second image component, and a third image component) and auxiliary information (such as first auxiliary information or second auxiliary information) of the image to be filtered are fused and input into the filtering network, the output may be only the first image component of the image to be filtered, or the second image component of the image to be filtered, or the third image component of the image to be filtered; it may also be the first image component and the second image component of the image to be filtered, or the second image component and the third image component of the image to be filtered, or the first image component and the third image component of the image to be filtered; or it may even be the first image component, the second image component, and the third image component of the image to be filtered; the embodiments of this application do not make specific limitations.

[0075] Taking the example of three image components of the image to be filtered being simultaneously input into the filtering network and using a cascaded processing structure of split-merge-split, see [reference needed]. Figure 7 This illustrates a schematic diagram of the composition structure of a loop filter framework 70 provided in an embodiment of this application. Figure 7As shown, the loop filtering framework 70 may include three image components (represented by Y, U, and V respectively) 701 of the image to be filtered, a first splitting unit 702, first auxiliary information 703, a first processing unit for the Y image component 704, a first processing unit for the U image component 705, a first processing unit for the V image component 706, second auxiliary information 707, an input fusion unit 708, a joint processing unit 709, a second splitting unit 710, a second processing unit for the Y image component 711, a second processing unit for the U image component 712, a second processing unit for the V image component 713, a first adder 714, a second adder 715, a third adder 716, and three filtered image components (represented by Out_Y, Out_U, and Out_V respectively) 717. Specifically, after the three image components 701 of the image to be filtered pass through the first splitting unit 702, they are divided into three signals: Y image component, U image component, and V image component. The first Y image component and its corresponding first auxiliary information 703 enter the Y image component first processing unit 704, the second U image component and its corresponding first auxiliary information 703 enter the U image component first processing unit 705, and the third V image component and its corresponding first auxiliary information 703 enter the V image component first processing unit 706, thus outputting three new image components. The input fusion unit 708 is used to fuse these three new image components and the second auxiliary information 707, and then input them to the joint processing unit 709. The joint processing unit 709 includes a multi-layer convolutional filtering network for performing convolution calculations on the input information. Since the specific convolution calculation process is similar to the related technical solutions, the specific execution steps of the joint processing unit 709 will not be described further. After passing through the joint processing unit 709, the signal will enter the second splitting unit 710 to be re-divided into three signals. These three signals are then input to the second processing unit 711 for the Y image component, the second processing unit 712 for the U image component, and the second processing unit 713 for the V image component, respectively, to obtain the residual information of the Y image component, the residual information of the U image component, and the residual information of the V image component. The Y image component of the three image components 701 of the image to be filtered, along with the obtained residual information of the Y image component, is input together to the first adder 714. The first adder 714... The output of step 4 is the filtered Y image component (represented by Out_Y); the residual information of the U image component in the three image components 701 of the image to be filtered and the obtained U image component are input together into the second adder 715, and the output of the second adder 715 is the filtered U image component (represented by Out_U); the residual information of the V image component in the three image components 701 of the image to be filtered and the obtained V image component are input together into the third adder 716, and the output of the third adder 716 is the filtered V image component (represented by Out_V).Here, regarding the output components, if only the filtered Y image component needs to be output, the loop filtering frame 70 may not include the second splitting unit 710, the second adder 715, and the third adder 716; if only the filtered U image component needs to be output, the loop filtering frame 70 may not include the second splitting unit 710, the first adder 714, and the third adder 716; if both the filtered Y image component and the filtered U image component need to be output, the loop filtering frame 70 may not include the third adder 716; the embodiments of this application do not impose specific limitations.

[0076] Taking the example of two image components of the image to be filtered being simultaneously input into the filtering network and using a split-and-merge cascaded processing structure, see [reference needed]. Figure 8 This illustrates a schematic diagram of the composition structure of another loop filter framework 80 provided in an embodiment of this application. Figure 8 As shown, the loop filtering framework 80 may include two image components (represented by Y and U respectively) 801 of the image to be filtered, a first splitting unit 702, first auxiliary information 703, a first processing unit 704 for the Y image component, a first processing unit 705 for the U image component, an input fusion unit 708, a joint processing unit 709, a second processing unit 711 for the Y image component, a first adder 714, and a filtered image component (represented by Out_Y) 802. Specifically, after the two image components 801 of the image to be filtered pass through the first splitting unit 702, they are divided into two signals: a Y image component and a U image component. The first Y image component and its corresponding first auxiliary information 703 enter the Y image component first processing unit 704, and the second U image component and its corresponding first auxiliary information 703 enter the U image component first processing unit 705, thus outputting two new image components. The input fusion unit 708 is used to fuse these two new image components and then input them to the joint processing unit 709. After passing through the joint processing unit 709, since only a single image component (i.e., the filtered Y image component) needs to be output, it does not need to enter the second splitting unit 710 at this time. It can be directly input to the Y image component second processing unit 711 to obtain the residual information of the Y image component. The Y image component of the two image components 801 of the image to be filtered and the residual information of the obtained Y image component are input together into the first adder 714. The output of the first adder 714 is the filtered Y image component (represented by Out_Y).

[0077] It should be noted that, since the size information of the Y image component can be different from that of the U or V image components, in Figure 7 The loop filter frame 70 shown is or Figure 8In the loop filtering framework 80 shown, an upsampling unit (or deconvolution unit or super-resolution unit) can be added before the first processing unit 705 for the U image component and the first processing unit 706 for the V image component to perform upsampling processing. This ensures that the resolution of the upsampled U image component or the upsampled V image component is consistent with that of the Y image component, facilitating subsequent loop filtering processing. Furthermore, with... Figure 7 Taking the loop filter framework 70 shown as an example, the improved loop filter in this embodiment may include at least an input fusion unit 708, a joint processing unit 709, and a first adder 714, a second adder 715, and a third adder 716. However, it may also include a first splitting unit 702, a first processing unit 704 for the Y image component, a first processing unit 705 for the U image component, a first processing unit 706 for the V image component, etc., and may even include a second splitting unit 710, a second processing unit 711 for the Y image component, a second processing unit 712 for the U image component, a second processing unit 713 for the V image component, etc. The embodiments in this application do not make specific limitations.

[0078] Furthermore, in the embodiments of this application, the loop filtering implementation method can adopt a split-merge-split processing structure, for example... Figure 7 The loop filter framework 70 shown can also employ a smaller split-and-comb processing structure, for example... Figure 8 The loop filter framework 80 shown can also employ fewer merging-debugging processing structures, or even fewer merging-debugging processing structures or more debugging-merging-debugging-merging-debugging processing structures. The embodiments of this application do not impose specific limitations.

[0079] It should also be noted that both the first and second auxiliary information can participate in the loop filtering process, for example... Figure 7 The loop filtering framework 70 shown; the first auxiliary information and the second auxiliary information can also selectively participate in the loop filtering process, for example... Figure 8 The loop filtering framework 80 shown in the diagram does not involve the second auxiliary information in the loop filtering process. In this embodiment, both the first and second auxiliary information may participate in the loop filtering process, or the first auxiliary information may not participate, or the second auxiliary information may not participate, or even neither the first nor the second auxiliary information may participate. This embodiment does not impose any specific limitations.

[0080] It should also be noted that the fusion stages of the first auxiliary information and the second auxiliary information can be the same or different; that is, the first auxiliary information and the second auxiliary information can participate in the loop filtering process in the same stage or in different stages. This application does not specifically limit this. For example, still using... Figure 7 Taking the loop filtering framework 70 shown as an example, both the first auxiliary information 703 and the second auxiliary information 707 can participate in the loop filtering process within the stage corresponding to the first splitting unit 702; or, both the first auxiliary information 703 and the second auxiliary information 707 can participate in the loop filtering process within the stage corresponding to the input fusion unit 708; or, both the first auxiliary information 703 and the second auxiliary information 707 can participate in the loop filtering process within the stage corresponding to the second splitting unit 710; or, the first auxiliary information 703 participates in the loop filtering process within the stage corresponding to the first splitting unit 702, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the input fusion unit 708; or, the first auxiliary information 703 participates in the loop filtering process within the first splitting unit 702. The first auxiliary information 703 participates in the loop filtering process before the stage corresponding to the first branching unit 702, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the second branching unit 710; or, the first auxiliary information 703 participates in the loop filtering process before the stage corresponding to the first branching unit 702, and the second auxiliary information 707 participates in the loop filtering process within the stage corresponding to the second branching unit 710; that is, the first auxiliary information 703 and the second auxiliary information 707 can flexibly select the fusion stage in the cascaded processing structure, and the embodiments of this application do not make specific limitations.

[0081] by Figure 7Taking the loop filtering framework 70 shown as an example, it uses a deep learning network (such as CNN) for loop filtering. The difference from the traditional CNN filter is that the improved loop filter in this embodiment adopts a cascaded processing structure, which can simultaneously input the three image components of the image to be filtered into the filtering network. It also incorporates other coding-related auxiliary information (such as block partitioning information, quantization parameter information, MV information, and other coding parameters). This auxiliary information can be incorporated into the filtering network at the same stage or at different stages. In this way, it not only makes full use of the relationship between the three image components, but also uses other coding-related auxiliary information to assist filtering, thereby improving the filtering quality. In addition, processing the three image components simultaneously effectively avoids the problem of needing to perform three complete network forward calculations for these three image components, thereby reducing computational complexity and saving coding bitrate. For example, based on VTM3.0, an experimental test found that, compared with related technical solutions, the loop filtering implementation method of this application embodiment can simultaneously achieve a 6.4% bit rate reduction for the Y image component, a 9.8% bit rate reduction for the U image component, and an 11.6% bit rate reduction for the V image component, under the premise of the same restored video quality, thereby saving coding bit rate.

[0082] The above embodiments provide a loop filtering implementation method. This method involves acquiring an image to be filtered, which is generated during the video encoding process from the original image in the video to be encoded. The video to be encoded includes original image frames, and each original image frame includes the original image. Based on the image to be filtered, at least two image components of the image to be filtered are obtained. Fusion information of the image to be filtered is determined, and the fusion information includes at least the information obtained by fusing the at least two image components. Based on the fusion information, a loop filtering process is performed on the image to be filtered to obtain at least one image component after filtering. Thus, by employing a cascaded processing structure and using encoding parameters such as block partitioning information and / or QP information as auxiliary information to fuse with multiple input image components, the method not only fully utilizes the relationships between multiple image components but also reduces computational complexity and saves encoding bitrate. Furthermore, it further improves the subjective and objective quality of the video reconstructed image during the encoding and decoding process.

[0083] Based on the same inventive concept as the foregoing embodiments, see [link to previous document]. Figure 9 This diagram illustrates the structural composition of a loop filtering implementation device 90 provided in an embodiment of this application. The loop filtering implementation device 90 may include: an acquisition unit 901, a splitting unit 902, a determination unit 903, and a filtering unit 904, wherein... The acquisition unit 901 is configured to acquire an image to be filtered; wherein the image to be filtered is generated during the video encoding process from the original image in the video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image; The splitting unit 902 is configured to obtain at least two image components of the image to be filtered based on the image to be filtered. The determining unit 903 is configured to determine the fusion information of the image to be filtered; wherein the fusion information includes at least the information obtained by fusing the at least two image components; The filtering unit 904 is configured to perform loop filtering on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered.

[0084] In the above scheme, the splitting unit 902 is specifically configured to perform component processing on each of the at least two original image components of the image to be filtered, based on at least two original image components of the image to be filtered, and obtain at least two image components of the image to be filtered after processing.

[0085] In the above scheme, the loop filtering implementation device 90 further includes a fusion unit 905, configured to perform fusion processing on at least two image components of the image to be filtered to obtain fusion information of the image to be filtered.

[0086] In the above scheme, the determining unit 903 is further configured to determine first auxiliary information corresponding to each original image component based on at least two original image components of the image to be filtered; wherein the first auxiliary information includes at least block partitioning information and / or quantization parameter information.

[0087] In the above scheme, the splitting unit 902 is further configured to add each original image component and the first auxiliary information corresponding to each original image component to the at least two original image components of the image to be filtered, and obtain at least two image components of the image to be filtered after processing.

[0088] In the above scheme, the fusion unit 905 is further configured to perform fusion processing on at least two image components of the image to be filtered and the first auxiliary information corresponding to each original image component to obtain the fusion information of the image to be filtered.

[0089] In the above scheme, the splitting unit 902 is further configured to obtain residual information corresponding to at least one of the at least two original image components based on the fusion information; The filtering unit 904 is specifically configured to sum the residual information corresponding to at least one of the at least two original image components and obtain at least one image component of the image to be filtered.

[0090] In the above scheme, the determining unit 903 is further configured to determine second auxiliary information corresponding to each original image component based on at least two original image components of the image to be filtered; wherein the second auxiliary information is different from the first auxiliary information.

[0091] In the above scheme, the fusion unit 905 is further configured to perform fusion processing on at least two image components of the image to be filtered and the second auxiliary information corresponding to each original image component to obtain the fusion information of the image to be filtered.

[0092] In the above scheme, the splitting unit 902 is further configured to add second auxiliary information to each of the at least two original image components of the image to be filtered, based on the at least two original image components of the image to be filtered, and obtain at least two image components of the image to be filtered.

[0093] In the above scheme, the acquisition unit 901 is specifically configured to perform video encoding processing on the original image in the video to be encoded, and use the generated reconstructed image as the image to be filtered; or, The acquisition unit 901 is specifically configured to perform video encoding processing on the original image in the video to be encoded to generate a reconstructed image; perform preset filtering processing on the reconstructed image, and use the preset filtered image as the image to be filtered.

[0094] In the above scheme, see [reference] Figure 9 The loop filtering implementation device 90 further includes a sampling unit 906, configured to select a low-resolution image component for at least two original image components of the image to be filtered; and to perform upsampling processing on the low-resolution image component.

[0095] Understandably, in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0096] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] Therefore, this embodiment provides a computer storage medium storing a loop filter implementation program, which, when executed by at least one processor, implements the steps of the method described in the foregoing embodiment.

[0098] Based on the composition of the aforementioned loop filter implementation device 90 and the computer storage medium, see [link to documentation]. Figure 10 This illustrates a specific hardware structure example of the loop filtering implementation device 90 provided in this application embodiment, which may include: a network interface 1001, a memory 1002, and a processor 1003; the various components are coupled together through a bus system 1004. It is understood that the bus system 1004 is used to implement the connection and communication between these components. In addition to a data bus, the bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 10 All buses are labeled as bus system 1004. Among them, network interface 1001 is used for receiving and sending signals during the process of sending and receiving information with other external network elements; The memory 1002 is used to store computer programs that can run on the processor 1003; Processor 1003, when running the computer program, performs the following: Obtain the image to be filtered; wherein, the image to be filtered is generated during the video encoding process from the original image in the video to be encoded, the video to be encoded includes original image frames, and the original image frames include the original image; Based on the image to be filtered, at least two image components of the image to be filtered are obtained respectively; Determine the fusion information of the image to be filtered; wherein the fusion information includes at least the information obtained by fusing the at least two image components; Based on the fusion information, the image to be filtered is subjected to loop filtering to obtain at least one image component of the filtered image.

[0099] It is understood that the memory 1002 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1002 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0100] The processor 1003 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1003 or by instructions in software form. The processor 1003 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1002. Processor 1003 reads the information in memory 1002 and completes the steps of the above method in conjunction with its hardware.

[0101] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, other electronic units for performing the functions described herein, or combinations thereof.

[0102] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0103] Alternatively, as another embodiment, the processor 1003 is also configured to perform the steps of the method described in the foregoing embodiments when running the computer program.

[0104] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0105] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0106] Industrial applicability In this embodiment, the image to be filtered is first obtained, which is generated from the original image in the video to be encoded during the video encoding process. Then, based on the image to be filtered, at least two image components of the image to be filtered are obtained respectively; this process can be regarded as a splitting stage. Next, the fusion information of the image to be filtered is determined; wherein, the fusion information includes at least the fusion of the at least two image components; this process can be regarded as a merging stage. In this way, by adopting a cascaded processing structure and fusing multiple input image components, not only is the relationship between multiple image components fully utilized, but the problem of having to perform multiple complete network forward calculations on these multiple image components is also effectively avoided, thereby reducing computational complexity and saving encoding bitrate. Finally, based on the fusion information, the image to be filtered is subjected to loop filtering to obtain at least one image component of the filtered image. In this way, the fusion information can further assist filtering, improving the subjective and objective quality of the video reconstructed image during the encoding and decoding process.

Claims

1. A loop filtering implementation device, applied to a decoding system, characterized in that, The device includes: Memory, used to store computer programs; A processor, configured to perform the following steps when running the computer program: Based on at least two image components of the image to be filtered, component processing is performed on each of the at least two image components to obtain at least two image components after component processing. At least two image components after component processing are fused together. Based on the fusion process, loop filtering is performed to obtain at least one image component of the image to be filtered. The at least two image components include a first image component and a second image component, and at least one image component of the filtered image to be filtered includes the filtered second image component.

2. The apparatus according to claim 1, characterized in that, The fusion process includes inputting the at least two image components at once.

3. The apparatus according to claim 1, characterized in that, The steps also include: Based on at least two image components of the image to be filtered, first auxiliary information corresponding to each image component is determined; wherein, the first auxiliary information includes at least block partitioning information and / or quantization parameter information.

4. The apparatus according to claim 3, characterized in that, The at least two image components of the image to be filtered are used to perform component processing on each of the at least two image components to obtain at least two component-processed image components, including: Based on at least two image components of the image to be filtered, each image component and its corresponding first auxiliary information are added to each of the at least two image components to obtain at least two image components of the image to be filtered.

5. The apparatus according to claim 3, characterized in that, The fusion process of at least two image components after component processing includes: The image to be filtered is fused together with at least two image components and the first auxiliary information corresponding to each image component to obtain the fused information of the image to be filtered.

6. The apparatus according to claim 1, characterized in that, The step of performing loop filtering based on the fusion processing to obtain at least one image component of the image to be filtered includes: Based on the fusion process, residual information corresponding to at least one of the at least two image components is obtained respectively; The residual information corresponding to at least one of the at least two image components is summed with the residual information of the at least one image component to obtain at least one image component of the filtered image.

7. The apparatus according to claim 1, characterized in that, The steps also include: Obtain the image to be filtered, wherein the image to be filtered includes a reconstructed image, or the image to be filtered includes a reconstructed image after a preset filtering process.

8. A loop filtering implementation device, applied to an encoding system, characterized in that, The device includes: Memory, used to store computer programs; A processor, configured to perform the following steps when running the computer program: Based on at least two image components of the image to be filtered, component processing is performed on each of the at least two image components to obtain at least two image components after component processing. At least two image components after component processing are fused together. Based on the fusion process, loop filtering is performed to obtain at least one image component of the image to be filtered. The at least two image components include a first image component and a second image component, and at least one image component of the filtered image to be filtered includes the filtered second image component.

9. An encoding system, characterized in that, The encoding system is configured to perform the following steps to output a bitstream: Based on at least two image components of the image to be filtered, component processing is performed on each of the at least two image components to obtain at least two image components after component processing. At least two image components after component processing are fused together. Based on the fusion process, loop filtering is performed to obtain at least one image component of the image to be filtered. The at least two image components include a first image component and a second image component, and at least one image component of the filtered image to be filtered includes the filtered second image component.

Images