Loop filtering, video encoding, video decoding methods, electronic equipment and media
The loop filtering method addresses scaling inconsistencies by classifying pixels and applying specific scale factors, enhancing video encoding and decoding quality by reducing errors and improving fidelity.
Patent Information
- Application Number
- KR1020247014874
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-12-14
- Filing Date
- 2022-12-12
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-12-12
AI Technical Summary
In-loop filtering technologies face issues with inconsistent scaling during picture reconstruction, leading to significant errors between reconstructed and original values, affecting filtering effectiveness.
A loop filtering method involving pixel classification and multi-scale factor adjustment is applied, where each pixel class receives a specific scale factor, reducing deviations and enhancing filtering accuracy.
The method improves picture quality by minimizing errors and ensuring consistent scaling across different pixel classes, resulting in higher fidelity and efficiency in video encoding and decoding processes.
Smart Images

Figure 112024048256049-PCT00050_ABST
Abstract
Description
Technology Field
[0001] This application relates to the field of picture processing technology, for example, loop filtering, video encoding, video decoding methods, electronic equipment, and media. Background Technology
[0002] In-loop filtering technology can improve picture compression efficiency, enhance picture encoding quality, and reduce decoding errors through multi-layer picture reconstruction. However, filtering models in these technologies are typically offline networks, and since there is no original picture to reference during actual use, problems arise where the scaling is too strong or too weak during the filtering process. Consequently, the error between the reconstructed value and the primitive value becomes relatively large, affecting the filtering effect.
[0003] [Technical Solution]
[0004] The present application provides loop filtering, video encoding, a video decoding method, electronic equipment, and a medium.
[0005] An embodiment of the present application provides a loop filtering method, said loop filtering method, said loop filtering method
[0006] A step of acquiring a first reconstructed video unit processed by a loop filter;
[0007] A step of performing pixel classification for the first reconstructed video unit;
[0008] A step of determining a scale factor corresponding to each class;
[0009] The step of obtaining a second reconstructed video unit by performing scale adjustment on the first reconstructed video unit according to the scale factor above; is included.
[0010] An embodiment of the present application further provides a video encoding method, said video encoding method, said video encoding method,
[0011] A step of obtaining a first reconstructed picture by performing filtering on the reconstructed picture of the current picture;
[0012] A step of obtaining a second reconstructed picture by performing filtering processing on the first reconstructed picture using a loop filtering method;
[0013] The method includes the step of identifying loop filtering control information of the loop filtering method in the bitstream based on the filtering process above, wherein the first reconstructed picture includes a first reconstructed video unit and the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined according to the loop filtering method.
[0014] An embodiment of the present application further provides a video decoding method, said video decoding method, said video decoding method,
[0015] A step of obtaining a bitstream containing an encoded video sequence and obtaining loop filtering control information related to a loop filtering method;
[0016] A step of obtaining a first reconstructed picture by performing filtering on the reconstructed picture of the current picture;
[0017] The method includes the step of obtaining a second reconstructed picture by performing filtering processing on the first reconstructed picture using a loop filtering method based on the loop filtering control information above;
[0018] Here, the first reconstructed picture includes a first reconstructed video unit, the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined according to the loop filtering method.
[0019] Embodiments of the present application further provide electronic equipment, said electronic equipment comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, and when the processor executes said program, the loop filtering method or video encoding method or video decoding method is implemented.
[0020] An embodiment of the present application further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the program is executed by a processor, the loop filtering method or video encoding method or video decoding method is implemented. Brief explanation of the drawing
[0021] FIG. 1 is a schematic diagram of loop filtering provided in one embodiment. FIG. 2 is a flowchart of a loop filtering method provided in one embodiment. FIG. 3 is a schematic diagram of a multi-scale factor operation provided in one embodiment. FIG. 4 is a flowchart of a video encoding method provided in one embodiment. FIG. 5 is a schematic diagram of an encoding unit framework provided in one embodiment. FIG. 6 is a schematic diagram of a video encoding process provided in one embodiment. FIG. 7 is a schematic diagram of a video encoding process provided in one embodiment. FIG. 8 is a schematic diagram of ALF filtering provided in one embodiment. FIG. 9 is a schematic diagram of another video encoding process provided in one embodiment. FIG. 10 is a schematic diagram of another video encoding process provided in one embodiment. FIG. 11 is a schematic diagram of another video encoding process provided in one embodiment. FIG. 12 is a schematic diagram of another video encoding process provided in one embodiment. FIG. 13 is a schematic diagram of another video encoding process provided in one embodiment. FIG. 14 is a flowchart of a video decoding method provided in one embodiment. FIG. 15 is a schematic diagram of a decoding unit framework provided in one embodiment. FIG. 16 is a schematic diagram of the structure of a loop filtering device provided in one embodiment. FIG. 17 is a schematic diagram of the structure of a video encoding device provided in one embodiment. FIG. 18 is a schematic diagram of the structure of a video decoding device provided in one embodiment. FIG. 19 is a schematic diagram of the hardware structure of an electronic facility provided in one embodiment. Specific details for implementing the invention
[0022] Compressed video in H.266, a next-generation video encoding standard also known as Versatile Video Coding (VVC), contains distortion effects such as blocking, ringing, color deviation, and picture blurring. To reduce the impact of these distortions on video quality, filtering can be performed on video frames using in-loop filtering techniques. FIG. 1 is a schematic diagram of loop filtering provided in one embodiment. As shown in FIG. 1, an input original picture can be processed using Luma Mapping With Chroma Scaling (LMCS), a Deblocking Filter (DBF), a Sample Adaptive Offset (SAO), and / or an Adaptive Loop Filter (ALF), and a reconstructed picture can be output. Here, LMCS can improve compression efficiency by reassigning codewords to information within the dynamic range; DBF can be used to reduce blocking effects; SAO can be used to improve ringing effects; and ALF can be used to reduce decoding errors.
[0023] In the embodiments of the present application, a Neural Networks (NN)-based filtering method may be used in the loop filtering process. NN filtering may be added as a new module or step to the loop filtering, or may replace the modules or steps of DBF and SAO to save transmission overhead and ensure picture quality; and picture performance may be further enhanced by ALF after NN filtering.
[0024] In the embodiments of the present application, scaling can be performed on a picture during a loop filtering process, for example, by adjusting the luminance and / or chroma of pixels within a video frame. Furthermore, if there are too many pixel points in a picture frame and fitting is performed using a single straight line, only the scaling requirements of some pixel points may be satisfied, which may lead to some adverse effects. For example, if a specific pixel point needs to be enhanced, the scale of that pixel point may be reduced based on the fitting result, thereby introducing more deviation between the reconstructed value and the raw value during the scaling process, resulting in degraded picture quality and affecting the filtering effect. In the embodiments of the present application, scale factors of different classes can be determined separately through pixel classification, thereby providing a flexible scaling method.
[0025] In an embodiment of the present application, a loop filtering method is provided, which may be applied mainly to an encoding unit or to a decoding unit, and in a video encoding or decoding process, filtering processing may be performed on a reconstructed picture in a video using the loop filtering method. FIG. 2 is a flowchart of a loop filtering method provided in one embodiment, and as shown in FIG. 2, the method provided in this embodiment includes the following steps.
[0026] In step (110), the first reconstructed video unit processed by the loop filter is obtained.
[0027] In this embodiment, the loop filter may refer to a model used to perform filtering processing on an input picture, and supports various filtering methods such as NN, LMCS, DBF, SAO, and / or ALF filtering. The video unit may be a picture of one frame or multiple frames of a video, or one or multiple slices. The first reconstructed video unit is obtained after processing through the loop filter.
[0028] In step (120), pixel classification is performed on the first reconstructed video unit.
[0029] In this embodiment, according to a set rule, the first reconstructed video unit can be divided into a plurality of classes, for example, according to the type of luminance, chroma, region, and / or loop filter of the pixels in the first reconstructed video unit, and then, instead of performing scaling using the same scale factor for all pixels in the first reconstructed video unit, scaling is performed using a corresponding scale factor for each different class, thereby ensuring that all pixels in the first reconstructed video unit can be reasonably scaled, thereby guaranteeing a filtering effect.
[0030] In step (130), a scale factor corresponding to each class is determined.
[0031] In this embodiment, the scale factor is a key parameter for performing scale control on the luminance and / or chroma of a pixel, and if the magnitude of the scale factor is different, the corresponding control amplitude is also different. The scale factor corresponding to each class may be the same or different. The scale factor corresponding to each class can be calculated and obtained based on the difference in pixel values before and after filtering of a pixel point within the corresponding class; for example, a straight line is obtained by fitting based on the difference in pixel values before and after filtering of a pixel point within the corresponding class, and the scale factor of the corresponding class is determined according to the slope of the straight line; it is to be explained that by performing fitting for each class to obtain a corresponding straight line, the diversity of scale factors can be improved and flexible scale control for different classes can be implemented; furthermore, in the decoding stage, instead of calculating and obtaining the scale factor corresponding to each class, the scale factor corresponding to each class can be obtained by parsing the encoded video data.
[0032] In step (140), scale adjustment is performed on the first reconstructed video unit according to the scale factor to obtain the second reconstructed video unit.
[0033] In this embodiment, scaling can be performed using corresponding scale factors for each different class, and the operation of performing scaling based on such pixel classification and the scale factors of each classification can be referred to as a Multi-Scale Factors (MSF) operation; additionally, the luminance and / or chroma components of pixel points within each class can be adjusted. MSF filtering can be applied to the entire filtered picture, or to a single or multiple slices of the picture, or independently to the luminance or chroma components. The scaled second reconstructed video unit can be used as a reference and basis for subsequent encoding.
[0034] FIG. 3 is a schematic diagram of a multi-scale factor operation provided in one embodiment. As shown in FIG. 3, the loop filter is an NN filter, and a first video reconstruction unit is obtained after being filtered by the NN filter; pixel classification for the first video reconstruction unit can be performed according to chroma (including blue color difference component Cb and red color difference component Cr) and / or luminance component, where a digital identifier is used to indicate the classification to which each pixel belongs, for example, a pixel with a digital identifier of "1" belongs to the same classification, FIG. 3 shows a total of 4 classifications from 1 to 4; then, for each classification, a corresponding scale factor is calculated and denoted as S1, S2, S3, and S4, respectively, and scale adjustment is performed on the pixels of the corresponding classification using the scale factor corresponding to each classification to obtain a second video reconstruction unit.
[0035] The loop filtering method of the present embodiment can be understood as a pixel classification-based multi-scale factor filtering method, wherein a second reconstructed video unit is obtained after performing scale adjustment on a first reconstructed video unit according to class, and the deviation between the reconstructed value and the raw value in the second reconstructed video unit is smaller, has less distortion, has higher picture quality, and possesses a superior filtering effect.
[0036] In one embodiment, the loop filter includes a neural network-based loop filter (i.e., an NN filter). In this embodiment, the NN filter analyzes data using a format based on deep learning and establishes a mapping from the distorted picture (or reconstructed picture) to the original picture to implement picture quality enhancement. The NN filter is generally an offline network, and since there is no original picture to reference during actual use, there may be a problem where the scale is too strong or too weak. However, in this embodiment, by performing pixel classification and introducing a scale factor for each class, the scale of the pixel points can be flexibly controlled. In the case of the NN filter, various data that must be input to the neural network, such as reconstructed picture samples, quantization parameter (QP) information, code unit (CU) division information, deblocking filtering information, prediction samples, etc., can be configured, and this data is input to the NN filter, and a first reconstructed video unit processed by the NN filter is output.
[0037] In one embodiment, the reconstructed video unit includes a reconstructed picture, or slice, or code block, or code tree block. Here, the reconstructed video unit may be a first reconstructed video unit or a second reconstructed video unit.
[0038] In one embodiment, the step of performing scale adjustment for the first reconstructed video unit according to the scale factor is,
[0039] A step of performing scaling on the luminance component of the first reconstructed video unit according to a scale factor (as shown in FIG. 3);
[0040] A step of performing scaling on the first chroma component (e.g., Cb) of the first reconstructed video unit according to a scale factor;
[0041] The method includes at least one step of performing scaling on the second chroma component (e.g., Cr) of the first reconstructed video unit according to a scale factor.
[0042] In one embodiment, the step of determining a scale factor corresponding to each class is,
[0043] Step (131), pixel points of the same class are integrated into the same coordinate diagram, wherein the horizontal coordinate of each pixel point is the output of the corresponding pixel point after passing through a loop filter, and the vertical coordinate is the input of the corresponding pixel point before passing through the loop filter;
[0044] Step (133), performing fitting on pixel points in a coordinate diagram based on the least squares method to obtain a straight line, and using the slope of the straight line as a scale factor corresponding to a corresponding class; includes.
[0045] In this embodiment, the scale factor of each class can be obtained through calculation and applied to the encoding unit. Taking FIG. 3 as an example, pixel classification is performed on the first reconstructed video unit after NN filtering, and for one of the classes, the output of the pixel point belonging to the corresponding class after NN filtering is set as the horizontal coordinate, and the input before NN filtering is set as the vertical coordinate, so that pixel points belonging to the same class among the first reconstructed video units are placed on the same coordinate diagram; then, a single straight line is obtained by fitting using the least squares method, and the slope of the straight line is the scale factor corresponding to the corresponding class. Scale factors can be obtained as many times as there are classes. The method of fitting the straight line using the least squares method makes the reconstructed picture processed by the loop filter closer to the original picture, thereby achieving the effect of applying the filtering scale to the original picture.
[0046] For the class-th classification, the scale factor The calculation formula is as follows.
[0047] ; Here is the number of pixel points of the class-th classification, and Is is, Is And, Is i It is the input to the NN filter of the i-th pixel point, and Is i It is the output of the NN filter at the i-th pixel point, and Is i It is the raw pixel value of the nth pixel point; The value of Related to, and in this embodiment, am.
[0048] In one embodiment, the step of determining a scale factor corresponding to each class includes reading loop filtering control information related to a loop filtering method and determining a scale factor corresponding to each class.
[0049] In this embodiment, the scale factor of each class can also be read from loop filtering control information related to the loop filtering method by parsing the upper-level syntax of the bitstream and can be applied to the decoding stage.
[0050] In one embodiment, the step of performing scale adjustment on a first reconstructed picture according to a scale factor is,
[0051] For each pixel point in each class, the difference between the output of the corresponding pixel point after passing through the loop filter and the input of the corresponding pixel point before passing through the loop filter is multiplied by a scale factor corresponding to the class, and the result of the multiplication is added to the input of the corresponding pixel point before passing through the loop filter to obtain the pixel value after the corresponding pixel point is adjusted; includes.
[0052] In this embodiment, the scale of NN filtering is consistently limited using a scale factor corresponding to each classification, and the formula for scale control is and, here is the input to the NN filter, and is the output of the NN filter, and class is Indicates the class number corresponding to, and is the scale factor corresponding to the class-th class.
[0053] In one embodiment, the pixel classification method is,
[0054] Performing pixel classification based on the luma component;
[0055] Performing pixel classification based on the first chroma component;
[0056] Performing pixel classification based on the second chroma component;
[0057] Performing pixel classification based on block classification mode;
[0058] Performing pixel classification based on pixel point regions;
[0059] Performing pixel classification based on the class of a neural network-based loop filter (NN filter);
[0060] Performing pixel classification based on the classification mode used by the Deblocking Filter (DBF);
[0061] Performing pixel classification based on the classification mode used by Sample Adaptive Offset (SAO) filtering;
[0062] Performing pixel classification through a neural network used for pixel classification; includes at least one of these.
[0063] In this embodiment, pixel classification may be performed on a luminance component, or pixel classification may be performed based on a block classification mode of ALF, and additionally, various classification methods may be further supported by considering factors such as the luminance component of the picture, pixel point area, NN filtering, DBF filtering, and SAO filtering.
[0064] For example, based on the block classification mode of the ALF, pixel classification is performed according to the directional factor and / or activity factor of each pixel block. For example, if there are 5 directional factors and 5 activity factors, the first reconstructed video unit can be divided into 25 classes according to 2 factors, the first reconstructed video unit can be divided into 5 classes according to the activity factor, and also the first reconstructed video unit can be divided into 5 classes according to the directional factor.
[0065] Also, for example, pixel classification is performed based on the luminance component region of a pixel point, and, for instance, the first reconstructed video unit is divided into 8 regions, i.e., 8 classes, such that the number of encoding units in each class is as equal as possible;
[0066] Also, for example, pixel classification is performed based on NN filters, and for instance, in a loop filtering process, different pixel blocks may select different NN filters, and each pixel block is classified according to the NN filter used for each block;
[0067] Also, for example, based on the classification of other filtering models, and according to the classification mode used by DBF and / or SAO filtering, e.g., the first reconstructed video unit can be divided into different classes;
[0068] Also, for example, pixel classification can be performed using a neural network, and a neural network used for pixel classification can be trained, the input includes information such as the pixel value of the first reconstructed video unit and the parameters of the loop filter, and the output is the result of pixel classification.
[0069] In addition, classification may be performed according to chroma components, for example, based on the chroma component region of a pixel point, and, for instance, the first reconstructed video unit is divided into 8 regions, i.e., 8 classes, such that the number of encoding units in each class is as equal as possible;
[0070] Also, for example, classification is performed according to at least one chroma component among Cb and Cr, and pixel points where Cb and / or Cr are the same are divided into one class;
[0071] Also, for example, by combining with pixel classification information of the luminance component, the first reconstructed video unit is divided into different classes according to the chroma component.
[0072] In one embodiment, the step of performing pixel classification based on a block classification mode is,
[0073] Step (1210), a step of acquiring a pixel block of the first reconstructed video unit;
[0074] Step (1220), a step of calculating the Laplacian gradient of each pixel block and calculating a direction factor and an activity factor according to the Laplacian gradient;
[0075] Step (1230), determining the class of pixel blocks according to a directional factor and / or an activity factor, and determining the class of all pixel points in the reconstructed video unit; is included.
[0076] In this embodiment, pixel classification is performed based on the Block Classification mode during ALF filtering, and according to the luminance component of the first reconstructed video unit, for example, a 4×4 pixel block (or chroma block), the Laplace operator gradients in the horizontal 0°, vertical 90°, 135°, and 45° directions of each pixel in the surrounding 8×8 pixel blocks centered on the block are calculated for each 4×4 pixel block. To reduce computational complexity, the gradients in the horizontal 0°, vertical 90°, 135°, and 45° directions are calculated only for some pixel points where both the horizontal and vertical coordinates are even or neither is even, and , , and You can obtain each of these;
[0077] The direction factor D reflects the direction of the gradient, and first and Calculate the ratio of the maximum and minimum values of the horizontal and vertical gradients according to After obtaining, the diagonal gradient value and Calculate the ratio of the maximum and minimum values of the diagonal gradient according to, is obtained, and then compared with the threshold values t1=2 and t2=4.5 respectively according to the rules below to obtain the directional factor D.
[0078] ① , In this case, D=0 and;
[0079] ② , In this case, D=1 and;
[0080] ③ , In this case, D=2 and;
[0081] ④ , In this case, D=3 and;
[0082] ⑤ , In this case, D=4; the activity factor A reflects the strength of the gradient, and and Depending on the method, it can be obtained by searching the table.
[0083] Then, the class identifier to which the 4×4 pixel block belongs can be represented as filtIdx=5*D+A.
[0084] It must be explained that when there are too many classes of pixels, it can lead to an increase in the transmission overhead of the video encoding loop filtering bitstream, and also, for some classes, the scaling operation may not yield performance gains or may even degrade encoding performance; therefore, embodiments of the present application further provide some class-based competitive decision mechanism and a scaling switch control strategy.
[0085] In one embodiment, the step of performing scale adjustment for the first reconstructed video unit according to the scale factor is,
[0086] Step (1401), for each pixel point of each class, calculate the loss between the pixel after scaling using a scale factor corresponding to the class and the raw pixel;
[0087] Step (1403), according to the loss corresponding to the pixel points of each class, scaling is performed on the pixel points of the corresponding class using a scale factor corresponding to some classes; includes.
[0088] In this embodiment, for a plurality of classes, it is possible to further determine which class's pixel points should be scaled based on the degree of distortion, i.e., the cost, between the scaled pixels and the raw pixels. Based on performing pixel classification for the first reconstructed video unit and determining the scale factor for each class, scale is performed on the pixel points of the corresponding class using each scale factor, and the cost corresponding to each class's pixel points is calculated. This allows for selecting classes with relatively small costs to perform scale, and not performing scale on classes with relatively large costs. For example, in the case of a class where the cost is higher than a threshold, scale is performed on the pixel points of the corresponding class using the corresponding scale factor; furthermore, for example, the costs of each class are sorted from smallest to largest, and scale is performed on the pixel points of the corresponding classes using the corresponding scale factor for the five classes with the smallest costs, thereby improving the flexibility of scale and picture quality.
[0089] In one embodiment, the step of performing scale adjustment for the first reconstructed video unit according to the scale factor is,
[0090] Step (1411), calculates a first loss between the picture and the original picture after scaling is performed for each class's pixel point using a single scale factor;
[0091] Step (1413), calculating a second loss between the picture and the original picture after scaling is performed on the pixel points of the corresponding class using a scale factor corresponding to each class;
[0092] Step (1415), depending on the first loss and the second loss, scale-up is performed on the first reconstructed video unit using a single scale factor, or scale-up is performed on the pixel points of the corresponding class using a scale factor corresponding to each class.
[0093] In this embodiment, scaling may be performed for each class by selecting and using a single scale factor, or scaling may be performed for pixel points of the corresponding class by using a scale factor corresponding to each class. For example, a single scale factor is determined for a first reconstructed video unit, and a first loss is calculated between the picture after scaling is performed on the first reconstructed video unit and the original picture using the scale factor; then, based on performing pixel classification on the first reconstructed video unit and determining the scale factor for each class, scaling is performed for pixel points of the corresponding class using each scale factor, and then a second loss is calculated between the adjusted picture of the first reconstructed video unit and the original picture, and by comparing the first loss and the second loss, scaling is performed according to the scaling method with relatively small loss, thereby improving the flexibility of scaling and picture quality.
[0094] In one embodiment, the single scale factor is a value set according to statistical information; or the single scale factor is determined according to the slope of a straight line obtained by fitting each pixel point in the first reconstructed video unit.
[0095] In this embodiment, the single scale factor may be obtained by calculating for the first reconstructed video unit, for example, by placing all pixel points of the first reconstructed video unit on the same coordinate diagram and fitting according to all pixel points to obtain a single straight line, and using the slope of said straight line as the single scale factor; the single scale factor may be a preset scale factor, for example, by setting one appropriate scale factor according to statistical information.
[0096] In one embodiment, the step of performing scale adjustment for the first reconstructed video unit according to the scale factor is,
[0097] Step (1421), calculating a second loss between the video unit and the raw video unit after scaling is performed on the pixel points of the corresponding class using a scale factor corresponding to each class;
[0098] Step (1422), fusing the classes of difference values of corresponding scale factors within a set range into one class;
[0099] Step (1423), determine the scale factor corresponding to each fused class;
[0100] Step (1424), calculating a third loss between the video unit and the raw video unit after scaling is performed on the pixel points of the corresponding fused class using a scale factor corresponding to each fused class;
[0101] Step (1425), according to the second loss and the third loss, scale adjustment is performed on pixel points of the corresponding class using a scale factor corresponding to each class, or scale adjustment is performed on pixel points of the corresponding fused class using a scale factor corresponding to each fused class; includes.
[0102] In this embodiment, scaling can be performed on pixel points of a corresponding class using a scale factor corresponding to each class, and each class can be fused to reduce the number of classes, decrease the amount of computation, and improve the efficiency and effectiveness of scaling. For example, based on performing pixel classification on a first reconstructed video unit and determining the scale factor for each class, scaling is performed on pixel points of a corresponding class using each scale factor to calculate a second loss; additionally, each class is fused; For example, in the case of some classes with similar scale factors (e.g., the difference between scale factors does not exceed a specified range), these classes can be fused into one class, and then the scale factor of each fused class (e.g., by placing the pixel points of each class fused into one class on the same coordinate diagram and fitting a straight line to calculate the slope) is calculated, and a third loss is calculated after scaling is performed on the pixel points of the corresponding fused class using the scale factor corresponding to each fused class, and by comparing the second loss and the third loss, scaling is performed according to the scaling method with relatively small loss, thereby improving the flexibility of scaling and picture quality.
[0103] In one embodiment, the step of performing scale adjustment for the first reconstructed video unit according to the scale factor is,
[0104] Step (1431), for each pixel point of each class, calculate the loss between the pixel after scaling using a scale factor corresponding to the class and the raw pixel;
[0105] Step (1432), according to the loss corresponding to the pixel point of each class, the scale control switch flag is used to identify the switch state of the scale control corresponding to each class;
[0106] Step (1433), according to each scale control switch flag, performs scale control on pixel points of the corresponding class using a scale factor corresponding to the class for which scale control is turned on; includes.
[0107] In this embodiment, the purpose of performing scale adjustment for a desired class can be achieved by controlling the scale adjustment switch of each class by setting a scale adjustment switch flag for each class.
[0108] For multiple classes, depending on the degree of distortion between the scaled pixels and the raw pixels, i.e., the cost, it can be further determined which class to turn on the scale and which class to turn off the scale. Based on performing pixel classification for the first reconstructed video unit and determining the scale factor for each class, scale is performed on the pixel points of the corresponding class using each scale factor, and the cost corresponding to the pixel points of each class is calculated. If there is no performance gain or even performance degradation after scaling a pixel point of one class, the scale switch flag for that class can be set to 0, which indicates that scale is not performed for that class; otherwise, the scale switch flag for that class can be set to 1, which indicates that scale is performed for that class. Based on this, scale can be performed using the corresponding scale factor for the class for which scale is turned on according to the scale switch flag, thereby improving the flexibility of scale and picture quality.
[0109] In one embodiment, the step of performing scale adjustment for the first reconstructed video unit according to the scale factor is,
[0110] Step (1441), calculates a first loss between the video unit and the raw video unit after scaling is performed for each class's pixel point using a single scale factor;
[0111] Step (1443), calculating a second loss between the video unit and the raw video unit after scaling is performed on the pixel points of the corresponding class using a scale factor corresponding to each class;
[0112] Step (1445), according to the first loss and the second loss, using a scale control switch flag, the switch state of scale control corresponding to a single scale factor and the switch state of scale control corresponding to each class are each identified;
[0113] Step (1447), depending on each scale control switch flag, scale control is performed for pixel points of each class using a single scale factor, or scale control is performed for pixel points of the corresponding class using a scale factor corresponding to the class for which scale control is turned on; includes.
[0114] In this embodiment, the scale control method can be controlled by setting a scale control switch flag for each class and setting a scale control switch flag for a single scale factor, and additionally, when scale control is performed for each class, the scale control turn-on or turn-off of each class can be further controlled.
[0115] For example, after determining a single scale factor for the first reconstructed video unit, the scale factor for each class is determined according to the pixel classification result, and for example, according to the activity factor A in ALF, the first reconstructed video unit can be divided into N classes, which correspond to N scale factors, and if a single scale factor is added, there are a total of N+1 scale factors; scale adjustment is performed on the first reconstructed video unit using the single scale factor to calculate the first loss; scale adjustment is performed on the pixel points of the corresponding classes using the N scale factors, and the second loss after adjusting the first reconstructed video unit is calculated by comparing the first loss and the second loss, and in the case of a scale adjustment method with relatively small loss, the scale adjustment switch flag is set to 1 to indicate that scale adjustment is turned on; in the case of a scale adjustment method with relatively large loss, the scale adjustment switch flag is set to 0 to indicate that scale adjustment is turned off. Here, when scaling is performed for pixel points of corresponding classes using N scale factors, a scaling switch flag can also be set for each class depending on whether performance is improved after the pixel points of each class are scaled.
[0116] The loop filtering method of the present embodiment provides a class-based competitive decision mechanism (e.g., competitive decision for scaling between each class, competitive decision for scaling between a single scale factor and multiple scale factors, competitive decision for each class scaling and fused class scaling) and a scale switch control strategy, thereby improving the flexibility of scaling, reducing the loss of scaling, and reducing the loss of the reconstructed video unit, thus ensuring the quality of the reconstructed picture and improving loop filtering performance.
[0117] In the embodiments of the present application, a video encoding method is further provided, and said video encoding method may be applied to the encoding stage. said video encoding method may be implemented based on a hybrid coding framework, for example, the next-generation video encoding standard H.266 / VVC encoding framework developed by the International Telecommunication Union Telecommunication Standardization Sector ITU-T and the Joint Video Project of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) includes modules such as intra-prediction, inter-prediction, transformation, quantization, loop filtering, and entropy encoding.
[0118] It should be noted that the method provided in the embodiments of the present application may be applied to the H.266 / VVC standard, the Audio Video Coding Standard (e.g., AVS3), or the next-generation video encoding and decoding standard, but the embodiments of the present application are not limited thereto.
[0119] FIG. 4 is a flowchart of a video encoding method provided in one embodiment. As illustrated in FIG. 4, the method provided in this embodiment is,
[0120] In step (210), filtering is performed on the reconstructed picture of the current picture to obtain a first reconstructed picture;
[0121] In step (220), filtering processing is performed on the first reconstructed picture using a loop filtering method to obtain a second reconstructed picture;
[0122] In step (230), loop filtering control information of the loop filtering method is identified in the bitstream based on the filtering process; comprising,
[0123] Here, the first reconstructed picture includes a first reconstructed video unit, and the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined by the loop filtering method according to any embodiment above. For example, the step of obtaining the second reconstructed picture by performing filtering on the first reconstructed picture using the loop filtering method is,
[0124] Step (2301), obtain a first reconstructed video unit processed by a loop filter;
[0125] Step (2303), performing pixel classification for the first reconstructed video unit;
[0126] Step (2305), determine the scale factor corresponding to each class;
[0127] Step (2307), scale adjustment is performed on the first reconstructed video unit according to the scale factor of each class to obtain the second reconstructed video unit; mainly includes.
[0128] FIG. 5 is a schematic diagram of an encoding unit framework provided in one embodiment. As shown in FIG. 5, the overall framework process of the encoding unit is as follows.
[0129] (1) Divide the current picture of the input video into encoding units;
[0130] (2) The divided encoding unit is transmitted to an intra prediction module or an inter prediction module to perform prediction encoding, wherein the intra prediction module is mainly used to remove spatial correlation of the picture, and the inter prediction module is mainly used to remove temporal correlation of the picture;
[0131] (3) Subtract the obtained prediction value from the original block to obtain the residual value, then perform transformation and quantization on the residual value to remove frequency domain correlation, and perform lossy compression on the data;
[0132] (4) Entropy encoding is performed on all encoding parameters and residual values to form a binary stream for storage or transmission, and the output data of the entropy encoding module is the compressed bitstream of the original video;
[0133] (5) A block reconstruction value is obtained by adding the predicted value and the inversely quantized and inversely transformed residual value to form a reconstruction picture, wherein the reconstruction picture can be processed by filtering (e.g., LMCS, DBF, SAO, ALF and / or NN) to obtain a first reconstruction picture;
[0134] (6) The first reconstructed picture is filtered using a loop filter to obtain the first reconstructed picture and stored in the picture cache as a reference picture.
[0135] (7) By performing video encoding using the second reconstructed picture, an encoded video sequence is obtained, and in the encoding process, loop filtering control information of the loop filtering method in the bitstream is identified based on the filtering process.
[0136] In the embodiment, to reduce distortion between the reconstructed picture and the original picture, NN filter and MSF filtering are introduced into the loop filtering process, which can be added as a new step or module or replace DBF and SAO in the original filtering process, thereby reducing transmission overhead and improving picture quality.
[0137] The NN filter analyzes data using a deep learning-based format and establishes a mapping from the distorted picture to the original picture to complete picture quality enhancement. This embodiment performs pixel classification in a loop filtering process and introduces a scale factor for each class, and can perform flexible scaling using a class-based competitive decision mechanism and a scale control switch control strategy.
[0138] FIG. 6 is a schematic diagram of a video encoding process provided in one embodiment. In this embodiment, for a first reconstructed picture filtered by NN, an MSF is applied to a hybrid coding framework for video encoding. As illustrated in FIG. 6, the MSF comprises two parts: pixel classification and scaling based on multiple scale factors. Various filters are supported in the video encoding process, for example, a reconstructed picture can be obtained using LMCS, and a first reconstructed picture is obtained by performing NN filtering on the reconstructed picture, and then pixel classification is performed based on the first reconstructed picture (for example, the block classification mode of ALF determines the classification to which a pixel block belongs by calculating gradients, directional factors and / or activity factors), and scaling is performed according to the scale factors of each class (in this process, a competing decision mechanism and a scaling switch control strategy can be used) to obtain a second reconstructed picture.
[0139] The video encoding method of the present embodiment performs loop filtering and scale adjustment based on multiple scale factors on a reconstructed picture to provide a high-quality reconstructed picture for video encoding and improve the quality of the encoded video. In addition, by identifying loop filtering control information of the loop filtering method in the bitstream, a basis for decoding the loop filtering method is provided.
[0140] In one embodiment, after obtaining a second reconstructed picture,
[0141] Step (2310), perform block classification on the second reconstructed picture;
[0142] Step (2313), performing adaptive loop filtering (ALF) on pixel points of each class in the second reconstructed picture; further includes.
[0143] FIG. 7 is a schematic diagram of a video encoding process provided in one embodiment. This embodiment applies MSF to a hybrid coding framework for video encoding. As illustrated in FIG. 7, MSF includes two parts: pixel classification and scaling based on multiple scale factors. Various filters are supported in the video encoding process, which are illustrated in FIG. 7 as examples of NN filtering and ALF filtering. The process further includes the step of performing NN filtering on the first reconstructed picture before performing MSF filtering on the first reconstructed picture among the first reconstructed pictures; and after obtaining the second reconstructed picture, ALF filtering may also be performed on the second reconstructed picture to further improve filtering performance and picture quality. Here, pixel classification may refer to the block classification mode of ALF.
[0144] FIG. 8 is a schematic diagram of ALF filtering provided in one embodiment. As illustrated in FIG. 8, the ALF filtering process is,
[0145] (1) Block classification, for example, dividing the second reconstructed picture into pixel blocks (or luma blocks) of size 4x4;
[0146] (2) Obtain a filter template, and for the classification result in (1), each class corresponds to a group of filter coefficients and corresponds to four filter templates, where the four filter templates are no transformation, diagonal transformation, vertical flip, and rotation transformation, and then determine the filter coefficient adjustment method for each 4x4 pixel block;
[0147] (3) After determining one 7×7 filter template (including a transformation method) for each 4×4 luminance block, filtering processing is performed on each pixel of the 4×4 pixel block, and finally, a reconstructed picture is output; mainly includes.
[0148] In one embodiment, after obtaining a second reconstructed picture,
[0149] Step (2320), performing adaptive loop filtering on pixel points of each class in the second reconstructed picture according to the classification result of pixel classification; further includes.
[0150] FIG. 9 is a schematic diagram of another video encoding process provided in one embodiment. As shown in FIG. 9, scaling of multiple scale factors can be performed based on block classification of ALF filtering, that is, pixel classification within MSF can be implemented using the block classification mode of ALF filtering. The LMCS-filtered reconstructed picture first performs NN filtering, and then, based on the classification result of ALF block classification, first performs scaling of multiple scale factors, and then determines the optimal filter usage method by selecting a filter template, and performs adaptive filtering. Based on this, by performing scaling and adaptive filtering using sufficient classification results without the need for secondary classification, encoding efficiency can be improved and encoding complexity can be reduced.
[0151] In one embodiment, before performing filtering processing on the first reconstructed picture using a loop filtering method,
[0152] Step (2210), perform block classification on the first reconstructed picture;
[0153] Step (2213), performing adaptive loop filtering on pixel points of each class in the first reconstructed picture according to the classification result of block classification; further includes.
[0154] FIG. 10 is a schematic diagram of another video encoding process provided in one embodiment. Various filters are supported in the video encoding process, and FIG. 10 illustrates NN filtering and ALF filtering as examples. It further includes performing NN filtering and ALF filtering on the first reconstructed picture before performing MSF filtering on the first reconstructed picture among the first reconstructed pictures, thereby further improving filtering performance and picture quality. MSF includes two parts: pixel classification and scale control based on multiple scale factors, where pixel classification may refer to the block classification mode of ALF. In this embodiment, scale control of multiple scale factors is placed after ALF filtering so that the gain obtained by scale control is not covered by ALF filtering, thereby sufficiently guaranteeing the validity of scale control and improving the quality of the reconstructed picture.
[0155] In one embodiment, before determining the scale factor corresponding to each class,
[0156] Step (2304), performing adaptive loop filtering on pixel points of each class among the first reconstructed pictures according to the classification result of pixel classification; further includes.
[0157] FIG. 11 is a schematic diagram of another video encoding process provided in one embodiment. As shown in FIG. 11, scaling of multiple scale factors can be performed based on block classification of ALF filtering, that is, pixel classification within MSF can be implemented using the block classification mode of ALF filtering. The LMCS-filtered reconstructed picture first performs NN filtering, then determines the optimal filter usage method by first selecting a filter template according to the classification result of ALF block classification, performs adaptive filtering, and then performs scaling of multiple scale factors. Based on this, without the need to perform secondary classification, adaptive filtering and scaling are performed by sufficiently utilizing the classification result, thereby improving encoding efficiency and reducing encoding complexity. Furthermore, since the gain obtained by scaling is not covered by ALF filtering, the validity of scaling is sufficiently guaranteed, and the quality of the reconstructed picture can be improved.
[0158] In one embodiment, before performing filtering processing on the first reconstructed picture using a loop filtering method,
[0159] Step (2220), performing DBF filtering on the reconstructed picture; further including;
[0160] After obtaining the second reconstructed picture,
[0161] Step (2330), performing SAO filtering on the second reconstructed picture; further includes.
[0162] FIG. 12 is a schematic diagram of another video encoding process provided in one embodiment. In this embodiment, when a loop filtering module such as DBF and / or SAO is turned on, MSF filtering may be added. As illustrated in FIG. 12, taking as an example where both DBF and SAO are turned on, NN filtering may be between DBF filtering and SAO filtering;
[0163] (1) After obtaining the reconstructed picture of the LMCS, first perform DBF filtering to obtain the first reconstructed picture;
[0164] (2) Perform an NN filtering operation to obtain a first reconstructed picture processed by a loop filter;
[0165] (3) MSF filtering is performed on the first reconstructed picture, comprising two parts: pixel classification and scaling of multiple scale factors, wherein the pixel classification may refer to the block classification mode of ALF;
[0166] (4) Perform SAO filtering on the MSF-filtered second reconstructed picture;
[0167] (5) Perform ALF filtering on the second reconstructed picture filtered by SAO, and output the final reconstructed picture.
[0168] In one embodiment, before performing filtering processing on the first reconstructed picture using a loop filtering method,
[0169] Step (2230), performs DBF filtering and SAO filtering on the reconstructed picture; further includes.
[0170] FIG. 13 is a schematic diagram of another video encoding process provided in one embodiment. In this embodiment, the loop filter may not use an NN filter, but may use a filtering model among related techniques such as DBF and / or SAO. Taking the case where both DBF and SAO are turned on as shown in FIG. 13 as an example,
[0171] (1) After obtaining the reconstructed picture of the LMCS, first perform DBF and SAO filtering to obtain the first reconstructed picture processed by the loop filter;
[0172] (2) MSF filtering is performed on the first reconstructed picture, comprising two parts: pixel classification and scaling of multiple scale factors, wherein the pixel classification may refer to the block classification mode of ALF;
[0173] (3) Perform ALF filtering on the second reconstructed picture filtered by MSF, and output the final reconstructed picture.
[0174] In the video encoding process of the above embodiment, the loop filtering method can be used in combination with other filtering, the order of filtering can be flexibly set, and classification can be performed once or twice, and MSF filtering can use different classification methods, scale factor competition strategies and switch control, thereby providing a high-quality reconstructed picture as a reference for video encoding and ensuring the quality of the encoded video.
[0175] In the embodiments of the present application, a video decoding method is further provided. Classification-based multi-scale factor adjustment is a loop filtering method, which can also be applied to video decoding.
[0176] FIG. 14 is a flowchart of a video decoding method provided in one embodiment, and as shown in FIG. 14, the method provided in this embodiment is,
[0177] In step (310), a bitstream containing an encoded video sequence is obtained, and loop filtering control information related to a loop filtering method is obtained;
[0178] In step (320), filtering is performed on the reconstructed picture of the current picture to obtain a first reconstructed picture;
[0179] In step (330), filtering processing is performed on the first reconstructed picture using a loop filtering method based on the loop filtering control information to obtain a second reconstructed picture; wherein
[0180] Here, the first reconstructed picture includes a first reconstructed video unit, and the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined according to the loop filtering method of any embodiment above. For example, the step of obtaining a second reconstructed picture by performing filtering processing on the first reconstructed picture using the loop filtering method is,
[0181] Step (3401), obtain a first reconstructed video unit processed by a loop filter;
[0182] Step (3403), performing pixel classification for the first reconstructed video unit;
[0183] Step (3405), determine the scale factor corresponding to each class;
[0184] Step (3407), scale adjustment is performed on the first reconstructed video unit according to the scale factor of each class to obtain the second reconstructed video unit; includes.
[0185] FIG. 15 is a schematic diagram of a decoding unit framework provided in one embodiment. The decoding framework includes modules or steps such as intra-prediction, motion compensation, inverse transformation, inverse quantization, and context-based adaptive binary arithmetic coding, wherein the type of scaling, the method of pixel classification, the scale factor, etc., can be obtained by parsing the upper-level syntax of the bitstream. As illustrated in FIG. 15, the overall framework process of the decoding unit is as follows.
[0186] (1) Decode header information for the bitstream of the encoded video sequence, and for the decoding of context-based adaptive binary arithmetic coding, obtain loop filtering control information related to the loop filtering method;
[0187] (2) Decode the bitstream of the encoded video sequence to obtain the predicted value and the residual value, perform inverse transform and inverse quantization on the residual value, and add the predicted value and the inverse quantized and inverse transformed residual value to obtain the block reconstruction value and form the reconstruction picture; additionally, the spatial correlation or temporal correlation of the picture can be removed through intra prediction or inter prediction;
[0188] (3) After performing filtering processing such as LMS, DBF, SAO and / or ALF on the reconstructed picture, obtain the first reconstructed picture;
[0189] (4) Filtering is performed on the first reconstructed picture using a loop filtering method based on loop filtering control information, and after the first reconstructed picture is filtered by the loop filter, the second reconstructed picture is obtained and stored in the picture cache as a reference;
[0190] (5) Obtain decoding video data using the second reconstructed picture.
[0191] In this embodiment, to reduce distortion between the reconstructed picture and the original picture, an NN filter is introduced in the loop filtering process. This can be added as a new step or module, or can replace the DBF and SAO in the original filtering process, thereby reducing transmission overhead and improving picture quality. Additionally, loop filtering control information can be read from the bitstream, providing a reliable basis for video decoding.
[0192] It should be noted that the video decoding method provided in this embodiment is merely an implementation method for multi-scale factor control based on classification, and actual applications include various optional embodiments.
[0193] The video decoding method of the present embodiment performs loop filtering and scale adjustment based on multiple scale factors on a reconstructed picture, which provides a high-quality reconstructed picture for video decoding and improves the quality of encoding and decoding data.
[0194] In one embodiment, the loop filtering control information is,
[0195] A sequence-level filtering switch flag of an encoded video sequence, used to instruct to turn on or off a loop filtering method for the video sequence where the current picture is located;
[0196] Filtering picture level flag used to instruct loop filtering control information to appear in picture header information or slice header information;
[0197] It includes at least one of loop filter structure information used to instruct a video unit to perform loop filtering.
[0198] In this embodiment, loop filtering control information related to MSF filtering, including information such as MSF filtering switches, the number of pixel classification classes, and scale factors, can be displayed in the bitstream of an encoded video sequence through a high-level syntax, and the decoder performs header information decoding on the input encoded video sequence and performs MSF filtering based on the acquired loop filtering control information.
[0199] Loop filtering control information includes a sequence level filtering switch flag, and depending on the sequence level filtering switch flag, it is possible to determine whether loop filtering is turned on for the current video sequence (whether MSF filtering is performed), and the sequence level filtering switch flag can be expressed in the sequence parameter set (SPS).
[0200] Table 1 shows the syntax elements of sequence-level filtering switch flags.
[0201] Table 1 Syntax elements of sequence-level filtering switch flags
[0202]
[0203] The meaning of the syntax elements in Table 1 is as follows. sps_msf_enabled_flag: If 1, it indicates that loop filtering is turned on (MSF is turned on) for the current video sequence, and if 0, it indicates that loop filtering is not turned on (MSF is turned on) for the current video sequence.
[0204] Loop filtering control information includes a filtering picture level flag, and based on the filtering picture level flag, the MSF filtering level, i.e., the video unit to which the loop filtering control information is applied, can be determined. This may be applied to a reconstructed picture or one or more slices of a reconstructed picture, and the filtering picture level flag is indicated in the Picture Parameter Set (PPS).
[0205] Table 2 shows the syntax elements of the filtering image level flags.
[0206] Table 2 Syntax elements of filtering image level flags
[0207]
[0208] The meaning of the syntax elements in Table 2 is as follows. pps_msf_info_in_ph_flag: If 1, it indicates that loop filtering control information appears in the Picture Header, and if 0, it indicates that loop filtering control information does not appear in the Picture Header but may appear in the Slice Header.
[0209] In the case of picture header information, if pps_msf_info_in_ph_flag is 1, it indicates that loop filtering control information appears in the picture header information, and depending on the picture header information, information such as the number of scale factors or classes, switch flags corresponding to each class, and scale factors of each class can be obtained.
[0210] Table 3 shows the syntax elements of the picture header information.
[0211] Table 3 Syntax elements of picture header information
[0212]
[0213] In the case of slice header information, if pps_msf_info_in_ph_flag is 0, loop filtering control information does not appear in the picture header information, but appears in the slice header information. Depending on the filtering picture level flag, information such as the scale factor or the number of classes, the switch flag corresponding to each class, and the scale factor of each class can be obtained.
[0214] When pps_msf_info_in_ph_flag is 0, it indicates that MSF-related information does not appear in the picture header information but may appear in the slice header information. Depending on the slice header information, information such as the scale factor or the number of classes, the switch flag corresponding to each class, and the scale factor of each class can be obtained.
[0215] Table 4 shows the syntax elements of the slice header information.
[0216] Table 4 Syntax elements of slice header information
[0217]
[0218] In one embodiment, the video unit includes a picture or video slice in the video.
[0219] In one embodiment, the loop filter structure information is,
[0220] A video unit filtering switch flag used to instruct the video unit to turn on or off the loop filtering method;
[0221] A video unit component switch flag used to indicate a video unit component to which loop filtering has been applied;
[0222] Video unit class count, used to indicate the total number of classes of video unit pixel classification;
[0223] Filtering switch flags for each class of video unit, used to instruct each class of video unit to turn on or off the loop filtering method;
[0224] A video unit scale factor used to indicate a scale factor corresponding to a video unit or a class of video units; comprising at least one of the following:
[0225] Here, the video unit component is,
[0226] Luma component of the video unit;
[0227] First chroma component of the video unit;
[0228] It includes at least one of the second chroma components of the video unit.
[0229] In this embodiment, the loop filter (or MSF filter) switch, number of classes, and scale factor of each class of the current picture or slice can be obtained according to the loop filter structure information, and scale adjustment can be performed for each class based on this information.
[0230] Table 5 shows the syntax elements for performing scaling for a single class based on loop filter structure information.
[0231] Table 5 Syntax elements for performing scaling for a single class based on loop filter structure information
[0232]
[0233] The meanings of the syntax elements in Table 5 are as follows.
[0234] msf_enabled_flag: if 1, indicate to turn on loop filtering (or MSF filtering); if 0, indicate not to turn on loop filtering (or MSF filtering); and if the corresponding syntax element does not exist, indicate that msf_enabled_flag is 0;
[0235] msf_scale: Displays the value of the scale factor.
[0236] Table 6 shows the syntax elements for performing scaling for multiple classes according to loop filter structure information.
[0237] Table 6 Syntax elements for performing scaling for multiple classes based on loop filter structure information
[0238]
[0239] The meanings of the syntactic elements in Table 6 are as follows.
[0240] msf_enabled_flag: 1 indicates turning on loop filtering (or MSF filtering), 0 indicates not turning on loop filtering (or MSF filtering), and if the corresponding syntax element does not exist, indicates that msf_enabled_flag is 0.
[0241] num_msf_class: If msf_structure() is located in picture header information, num_msf_class indicates the number of classes of the current picture, and if msf_structure() is located in slice header information, num_msf_class indicates the number of classes of the current slice.
[0242] msf_scale[i]: Displays the value of the scale factor of the i-th class.
[0243] Table 7 shows syntax elements for performing scaling on multiple classes of luminance and chroma components, respectively, according to loop filter structure information.
[0244] Syntax elements for performing scaling on multiple classes of luminance and chroma components respectively, according to the loop filter structure information in Table 7.
[0245]
[0246] The meanings of the syntactic elements in Table 7 are as follows.
[0247] msf_enabled_flag: 1 indicates that loop filtering (or MSF filtering) is enabled, 0 indicates that loop filtering (or MSF filtering) is disabled, and if the corresponding syntax element does not exist, msf_enabled_flag is 0.
[0248] num_msf_class_luma: If msf_structure() is located in picture header information, num_msf_class_luma indicates the number of luma classes of the current picture MSF, and if msf_structure() is located in slice header information, num_msf_class_luma indicates the number of luma classes of the current slice MSF.
[0249] msf_scale_luma[ i ]: Displays the value of the scale factor of the i-th class of lumas.
[0250] msf_cb_enabled_flag: 1 indicates that Chroma Cb uses loop filtering (or MSF filtering), and 0 indicates that it does not use it. If the corresponding syntax element does not exist, it indicates that msf_cb_enabled_flag is 0.
[0251] msf_cr_enabled_flag: If 1, indicates that Chroma Cr uses loop filtering (or MSF filtering), and if 0, indicates that it does not use. If the corresponding syntax element does not exist, it indicates that msf_cr_enabled_flag is 0.
[0252] num_msf_class _chroma: If msf_structure() is located in picture header information, num_msf_class_luma indicates the number of chroma classes of the current picture MSF, and if msf_structure() is located in slice header information, num_msf_class_luma indicates the number of chroma classes of the current slice MSF.
[0253] msf_scale_Cb[i]: Displays the scale factor value of the i-th class of chroma Cb.
[0254] msf_scale_Cr[i]: Displays the scale factor value of the i-th class of chroma Cr.
[0255] Table 8 shows the syntax elements for performing scaling on the luminance and chroma components of some classes, respectively, according to the loop filter structure information.
[0256] Table 8 Syntax elements for performing scaling on the luminance and chroma components of some classes, respectively, based on loop filter structure information
[0257]
[0258] msf_enabled_flag: 1 indicates that loop filtering (or MSF filtering) is enabled, 0 indicates that it is disabled, and if the corresponding syntax element does not exist, msf_enabled_flag is 0.
[0259] num_msf_class_luma: If msf_structure() is located in picture header information, num_msf_class_luma indicates the number of luma classes of the current picture MSF, and if msf_structure() is located in slice header information, num_msf_class_luma indicates the number of chroma classes of the current slice MSF.
[0260] msf_class_enabled_flag _luma[i]: 1 indicates that the i-th class of lumas uses loop filtering (or MSF filtering), and 0 indicates that it does not use it.
[0261] msf_scale_luma[i]: Displays the value of the scale factor of the i-th class of lumas.
[0262] msf_cb_enabled_flag: 1 indicates that Chroma Cb uses MSF, and 0 indicates that it does not use it. If the corresponding syntax element does not exist, msf_cb_enabled_flag is 0.
[0263] msf_cr_enabled_flag: If 1, indicates that Chroma Cr uses MSF, and if 0, indicates that it does not use. If the corresponding syntax element does not exist, it indicates that msf_cr_enabled_flag is 0.
[0264] num_msf_class _chroma: Identifies the number of chroma classes of the current picture MSF.
[0265] msf_class_enabled_flag _chroma[i]: If msf_structure() is located in picture header information, num_msf_class_luma displays the number of chroma classes of the current picture MSF, and if msf_structure() is located in slice header information, num_msf_class_luma displays the number of chroma classes of the current slice MSF.
[0266] msf_scale_Cb[i]: Displays the scale factor value of the i-th class of chroma Cb.
[0267] msf_scale_Cr[i]: Displays the scale factor value of the i-th class of chroma Cr.
[0268] In one embodiment, the step of performing filtering on a video unit to which loop filtering is applied using a loop filtering method based on loop filtering control information is,
[0269] Step (3410), according to the video unit filtering switch flag, the loop filtering method identifies the video unit that has been turned on;
[0270] Step (3413), depending on the video unit component switch flag and each class filtering switch flag of the video unit, identify the video unit component to which loop filtering is applied and the class to which the loop filtering method is turned on in the video unit;
[0271] Step (3415), according to the video unit scale factor, using a loop filtering method to perform scaling on the video unit components of the class to which the loop filtering method is turned on; includes.
[0272] In this embodiment, based on loop filtering control information, scaling can be performed on the video unit components to which loop filtering is applied (e.g., luminance components, Cr and / or Cb) of the pixel points of the class to which the loop filtering method is turned on, using MSF filtering for the video unit to which the loop filtering method is turned on.
[0273] In one embodiment, after obtaining a second reconstructed picture,
[0274] Step (3420), performing adaptive loop filtering on pixels of each class in the second reconstructed picture according to the classification result of pixel classification; further includes.
[0275] In this embodiment, the LMCS-filtered reconstructed picture can first perform NN filtering, and then, based on the classification result of pixel classification (which may refer to the block classification mode of ALF), perform scaling of multiple scale factors. After that, by selecting a filter template, an optimal filter usage method is determined and adaptive filtering is performed, as can be seen in FIG. 9. Based on this, without the need to perform secondary classification, scaling and adaptive filtering are performed by sufficiently utilizing the classification result, thereby improving encoding efficiency and reducing encoding complexity.
[0276] In one embodiment, before performing filtering processing on the first reconstructed picture using a loop filtering method based on loop filtering control information,
[0277] Step (3310), perform block classification for the first reconstructed video unit;
[0278] Step (3313), performing adaptive loop filtering on pixels of each class in the first reconstructed picture according to the classification result of block classification; further includes.
[0279] In this embodiment, before performing MSF filtering on the first reconstructed picture among the first reconstructed pictures, NN filtering and ALF filtering may also be performed on the first reconstructed picture to further improve filtering performance and image quality, and refer to FIG. 10. MSF includes two parts: pixel classification and scale control based on multiple scale factors, where pixel classification may refer to the block classification mode of ALF. In this embodiment, scale control of multiple scale factors is placed after ALF filtering so that the gain obtained by scale control is not covered by ALF filtering, thereby sufficiently ensuring the validity of scale control and improving the quality of the reconstructed picture.
[0280] In one embodiment, before determining the scale factor corresponding to each class,
[0281] Step (3404), perform adaptive loop filtering on pixels of each class among the video units according to the classification result of pixel classification; further includes.
[0282] In this embodiment, the LMCS-filtered reconstructed picture can first perform NN filtering, then, based on the classification result of pixel classification (which may refer to the block classification mode of ALF), first select a filter template to determine the optimal filter usage method, perform adaptive filtering, and then perform scale adjustment of multiple scale factors, as can be seen in FIG. 11. Based on this, scale adjustment and adaptive filtering can be performed using the classification result sufficiently without the need to perform secondary classification, and the gain validity of scale adjustment can also be guaranteed.
[0283] In one embodiment, before obtaining a second reconstructed picture by performing filtering processing on a first reconstructed picture using a loop filtering method based on loop filtering control information,
[0284] Step (3320), performing DBF filtering on the reconstructed picture; further including;
[0285] After obtaining the second reconstructed picture,
[0286] Step (3430), performing SAO filtering on the decoded reconstructed picture; further includes.
[0287] In this embodiment, when a loop filtering module such as DBF and / or SAO is turned on, MSF filtering may be added, and the loop filtering process can be referenced in FIG. 12.
[0288] In one embodiment, before performing filtering processing on the first reconstructed picture using a loop filtering method based on loop filtering control information,
[0289] Step (3330), performs DBF filtering and SAO filtering on the reconstructed picture; further includes.
[0290] In this embodiment, the loop filter may use a filter among related technologies instead of an NN filter, and the loop filtering process may refer to FIG. 13.
[0291] In the video decoding process of the above embodiment, the loop filtering method can be used in combination with other filtering, the order of filtering can be flexibly set, and classification can be performed once or twice, and MSF filtering can use different classification methods, scale factor competition strategies and switch control, thereby providing a high-quality reconstructed picture as a reference for video decoding and ensuring the quality of the decoded video.
[0292] Embodiments of the present application further provide a loop filtering device. FIG. 16 is a structural schematic diagram of a loop filtering device provided in one embodiment. As shown in FIG. 16, the loop filtering device comprises:
[0293] A video unit acquisition module (410) configured to acquire a first reconstructed video unit processed by a loop filter;
[0294] A classification module (420) configured to perform pixel classification for the first reconstructed video unit;
[0295] A determination module (430) configured to determine a scale factor corresponding to each class;
[0296] It includes a control module (440) configured to obtain a second reconstructed video unit by performing scale adjustment on the first reconstructed video unit according to the scale factor.
[0297] The loop filtering device of the present embodiment obtains a second reconstructed video unit by performing scale adjustment on a first reconstructed video unit according to class, thereby having a smaller deviation between the reconstructed value and the raw value in the second reconstructed video unit, less distortion, higher picture quality, and a superior filtering effect.
[0298] In one embodiment, the loop filter includes a neural network-based loop filter.
[0299] In one embodiment, the reconstructed video unit includes a reconstructed picture, or slice, or code block, or code tree block.
[0300] In one embodiment, the control module (440) is,
[0301] Performing scale adjustment on the luminance component of the first reconstructed video unit according to the scale factor;
[0302] Performing scale adjustment on the first chroma component of the first reconstructed video unit according to a scale factor;
[0303] It consists of at least one of performing scale control on the second chroma component of the first reconstructed video unit according to a scale factor.
[0304] In one embodiment, the decision module (430) is,
[0305] A consolidation unit configured to consolidate pixel points of the same class into the same coordinate diagram—wherein the horizontal coordinate of each pixel point is the output of the corresponding pixel point after passing through a loop filter, and the vertical coordinate is the input of the corresponding pixel point before passing through the loop filter—;
[0306] It includes a fitting unit configured to obtain a straight line by performing fitting on pixel points in a coordinate diagram based on the least squares method, and to use the slope of the straight line as a scale factor corresponding to a corresponding class.
[0307] In one embodiment, the determination module (430) is configured to read loop filtering control information related to a loop filtering method and determine a scale factor corresponding to each class.
[0308] In one embodiment, the adjustment module (440) is configured to multiply the difference between the output of the corresponding pixel point after passing through the loop filter and the input of the corresponding pixel point before passing through the loop filter for each pixel point in each class by a scale factor corresponding to the class, and to add the result of the multiplication to the input of the corresponding pixel point before passing through the loop filter to obtain the pixel value after the corresponding pixel point is adjusted.
[0309] In one embodiment, the pixel classification method is,
[0310] Performing pixel classification based on the luma component;
[0311] Performing pixel classification based on the first chroma component;
[0312] Performing pixel classification based on the second chroma component;
[0313] Performing pixel classification based on block classification mode;
[0314] Performing pixel classification based on pixel point regions;
[0315] Performing pixel classification based on the class of a neural network-based loop filter;
[0316] Performing pixel classification based on the classification mode used by DBF;
[0317] Performing pixel classification based on the classification mode used by SAO filtering;
[0318] Performing pixel classification through a neural network used for pixel classification; includes at least one of these.
[0319] In one embodiment, performing pixel classification based on a block classification mode is,
[0320] Acquiring a pixel block of the first reconstructed video unit;
[0321] Calculating the Laplacian gradient of each pixel block, and calculating the direction factor and activity factor based on the Laplacian gradient;
[0322] Determining the class of pixel blocks based on directional factors and / or activity factors, and determining the class of all pixel points in the reconstructed video unit; includes.
[0323] In one embodiment, the control module (440) is,
[0324] A first calculation unit configured to calculate, for each pixel point of each class, the loss between the pixel after scaling using a scale factor corresponding to the class and the raw pixel;
[0325] It includes a first adjustment unit configured to perform scaling on pixel points of a corresponding class using a scale factor corresponding to some classes according to the loss corresponding to pixel points of each class.
[0326] In one embodiment, the control module (440) is,
[0327] A second calculation unit configured to calculate a first loss between the picture and the original picture after performing scaling for pixel points of each class using a single scale factor;
[0328] A third calculation unit configured to calculate a second loss between the picture and the original picture after performing scaling on pixel points of the corresponding class using a scale factor corresponding to each class;
[0329] A second adjustment unit configured to perform scaling for a first reconstructed video unit using a single scale factor according to a first loss and a second loss, or to perform scaling for pixel points of a corresponding class using a scale factor corresponding to each class, respectively;
[0330] In one embodiment, the single scale factor is a value set according to statistical information; or the single scale factor is determined according to the slope of a straight line obtained by fitting to each pixel point in the first reconstructed video unit.
[0331] In one embodiment, the control module (440) is,
[0332] A fourth calculation unit configured to calculate a second loss between a video unit and a raw video unit after performing scaling on pixel points of a corresponding class using a scale factor corresponding to each class;
[0333] A fusion unit configured to fuse classes of difference values of corresponding scale factors within a set range into a single class;
[0334] A determination unit configured to determine a scale factor corresponding to each fused class;
[0335] A fifth calculation unit configured to calculate a third loss between a video unit and a raw video unit after performing scaling on pixel points of a corresponding fused class using a scale factor corresponding to each fused class;
[0336] A third adjustment unit configured to perform scaling on pixel points of a corresponding class using a scale factor corresponding to each class according to the second loss and the third loss, or to perform scaling on pixel points of a corresponding fused class using a scale factor corresponding to each fused class.
[0337] In one embodiment, the control module (440) is,
[0338] A sixth calculation unit configured to calculate, for each pixel point of each class, the loss between the pixel after scaling using a scale factor corresponding to the class and the raw pixel;
[0339] A first identification unit configured to identify the switch state of scale control corresponding to each class using a scale control switch flag according to the loss corresponding to the pixel point of each class;
[0340] It includes a fourth adjustment unit configured to perform scaling on pixel points of a corresponding class using a scale factor corresponding to the class for which scaling is turned on, according to each scale adjustment switch flag.
[0341] In one embodiment, the control module (440) is,
[0342] A seventh calculation unit configured to calculate a first loss between a video unit and a raw video unit after performing scaling for pixel points of each class using a single scale factor;
[0343] An eighth calculation unit configured to calculate a second loss between a video unit and a raw video unit after performing scaling on pixel points of a corresponding class using a scale factor corresponding to each class;
[0344] A second identification unit configured to identify, respectively, a switch state of scale control corresponding to a single scale factor and a switch state of scale control corresponding to each class using a scale control switch flag according to a first loss and a second loss;
[0345] It includes a fifth adjustment unit configured to perform scaling for pixel points of each class using a single scale factor according to each scale adjustment switch flag, or to perform scaling for pixel points of a corresponding class using a scale factor corresponding to the class for which scaling is turned on.
[0346] The loop filtering device provided in this embodiment and the loop filtering method provided in this embodiment belong to the same inventive concept, and technical details not described in detail in this embodiment may be referred to in any of the above embodiments, and this embodiment has the same beneficial effect as performing the loop filtering method.
[0347] Embodiments of the present application further provide a video encoding device. FIG. 17 is a structural schematic diagram of a video encoding device provided in one embodiment. As shown in FIG. 17, the video encoding device comprises:
[0348] A filtering module (510) configured to perform filtering on a reconstructed picture of the current picture to obtain a first reconstructed picture;
[0349] A loop filtering module (520) configured to obtain a second reconstructed picture by performing filtering on the first reconstructed picture using a loop filtering method;
[0350] An encoding module (530) configured to identify loop filtering control information of the loop filtering method in the bitstream based on the filtering process above; comprising,
[0351] Here, the first reconstructed picture includes a first reconstructed video unit, the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined according to the loop filtering method of any embodiment above.
[0352] The video encoding device of the present embodiment performs loop filtering and scale adjustment based on multiple scale factors on a reconstructed picture to provide a high-quality reconstructed picture for video encoding and improve the quality of the encoded video. In addition, by identifying loop filtering control information of the loop filtering method in the bitstream, a basis for video decoding using the loop filtering method is provided.
[0353] In one embodiment, the device is,
[0354] A first block classification module configured to perform block classification on the second reconstructed picture after obtaining the second reconstructed picture;
[0355] It further includes a first ALF module configured to perform adaptive loop filtering on pixel points of each class in the second reconstructed picture.
[0356] In one embodiment, the device is,
[0357] It further includes a second ALF module configured to perform adaptive loop filtering on pixel points of each class among the second reconstructed pictures according to the classification result of pixel classification after obtaining the second reconstructed picture.
[0358] In one embodiment, the device is,
[0359] A second block classification module configured to perform block classification on the first reconstructed picture before performing filtering processing on the first reconstructed picture using a loop filtering method;
[0360] It further includes a third ALF module configured to perform adaptive loop filtering on pixel points of each class among the first reconstructed pictures according to the classification result of block classification.
[0361] In one embodiment, the device is,
[0362] It further includes a fourth ALF module configured to perform adaptive loop filtering on pixel points of each class in the first reconstructed picture according to the classification result of pixel classification before determining the scale factor corresponding to each class.
[0363] In one embodiment, the device is,
[0364] A DBF module configured to perform DBF filtering on a reconstructed picture before performing filtering processing on a first reconstructed picture using a loop filtering method;
[0365] It further includes an SAO module configured to perform SAO filtering on the second reconstructed picture after obtaining the second reconstructed picture.
[0366] In one embodiment, the device is,
[0367] It further includes a dual filtering module configured to perform DBF filtering and SAO filtering on the reconstructed picture before performing filtering processing on the first reconstructed picture using a loop filtering method.
[0368] The video encoding device provided in this embodiment and the video encoding method provided in this embodiment belong to the same inventive concept, and technical details not described in detail in this embodiment may be referred to in any of the above embodiments, and this embodiment has the same beneficial effect as performing the video encoding method.
[0369] Embodiments of the present application further provide a video decoding device. FIG. 18 is a schematic diagram of the structure of a video decoding device provided in one embodiment. As shown in FIG. 18, the video decoding device comprises:
[0370] An information acquisition module (610) configured to acquire a bitstream containing an encoded video sequence and to acquire loop filtering control information related to a loop filtering method;
[0371] A filtering module (620) configured to perform filtering on the reconstructed picture of the current picture to obtain a first reconstructed picture;
[0372] A loop filtering module (630) configured to obtain a second reconstructed picture by performing filtering processing on the first reconstructed picture using a loop filtering method based on the loop filtering control information above; wherein
[0373] Here, the first reconstructed picture includes a first reconstructed video unit, the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined according to the loop filtering method of any embodiment.
[0374] The video decoding device of the present embodiment performs loop filtering and scale adjustment based on multiple scale factors on a reconstructed picture to provide a high-quality reconstructed picture for video decoding and improve the quality of the decoded video data. In addition, loop filtering control information can be read from the bitstream, providing a reliable basis for video decoding.
[0375] In one embodiment, the loop filtering control information is,
[0376] A sequence-level filtering switch flag of an encoded video sequence, used to instruct to turn on or off a loop filtering method for the video sequence where the current picture is located;
[0377] Filtering picture level flag used to instruct loop filtering control information to appear in picture header information or slice header information;
[0378] It includes at least one of loop filter structure information used to instruct a video unit to perform loop filtering.
[0379] In one embodiment, the video unit includes a picture or video slice in the video.
[0380] In one embodiment, the loop filter structure information is,
[0381] A video unit filtering switch flag used to instruct the video unit to turn on or off the loop filtering method;
[0382] A video unit component switch flag used to indicate a video unit component to which loop filtering has been applied;
[0383] Video unit class count, used to indicate the total number of classes of video unit pixel classification;
[0384] Filtering switch flags for each class of video unit, used to instruct each class of video unit to turn on or off the loop filtering method;
[0385] A video unit scale factor used to indicate a scale factor corresponding to a video unit or a class of video units; comprising at least one of the following:
[0386] Here, the video unit component is,
[0387] Luma component of the video unit;
[0388] First chroma component of the video unit;
[0389] It includes at least one of the second chroma components of the video unit.
[0390] In one embodiment, the loop filter module (640) is,
[0391] A first identification unit configured to identify a video unit turned on by a loop filtering method according to a video unit filtering switch flag;
[0392] A second identification unit configured to identify a video unit component to which loop filtering is applied and a class to which the loop filtering method is turned on in the video unit, according to a video unit component switch flag and each class filtering switch flag of the video unit;
[0393] It includes a control unit configured to perform scale adjustment on the video unit components of the class to which the loop filtering method is turned on, using a loop filtering method according to the video unit scale factor.
[0394] In one embodiment, the device is,
[0395] It further includes a fifth ALF module configured to perform adaptive loop filtering on pixels of each class in the second reconstructed picture according to the classification result of pixel classification after obtaining the second reconstructed picture.
[0396] In one embodiment, the device is,
[0397] A second block classification module configured to perform block classification on a first reconstructed video unit before performing filtering processing on a first reconstructed picture using a loop filtering method based on loop filtering control information;
[0398] It further includes a 6th ALF module configured to perform adaptive loop filtering on pixels of each class among the 1st reconstructed picture according to the classification result of block classification.
[0399] In one embodiment, the device is,
[0400] It further includes a seventh ALF module configured to perform adaptive loop filtering on pixels of each class among the video units according to the classification result of pixel classification before determining the scale factor corresponding to each class.
[0401] In one embodiment, the device is,
[0402] A DBF module configured to perform DBF filtering on a reconstructed picture before performing filtering processing on a first reconstructed picture using a loop filtering method based on loop filtering control information;
[0403] It further includes an SAO module configured to perform SAO filtering on the decoded reconstructed picture after obtaining the second reconstructed picture.
[0404] In one embodiment, the device further includes a dual filtering module configured to perform DBF filtering and SAO filtering on a reconstructed picture before performing filtering processing on a first reconstructed picture using a loop filtering method based on loop filtering control information.
[0405] The video decoding device provided in this embodiment and the video decoding method provided in this embodiment belong to the same inventive concept, and technical details not described in detail in this embodiment may be referred to in any of the above embodiments, and this embodiment has the same beneficial effect as performing the video decoding method.
[0406] Embodiments of the present application further provide electronic equipment, said electronic equipment may be picture processing equipment, encoding unit equipment, or decoding unit equipment. FIG. 19 is a schematic diagram of the hardware structure of electronic equipment provided in one embodiment, and as shown in FIG. 19, the electronic equipment provided in the present application includes a memory (720), a processor (710), and a computer program stored in the memory and executable by the processor, and when the processor (710) executes said program, it implements the loop filtering method, the video encoding method, or the video decoding method.
[0407] The electronic equipment may further include a memory (720); the processor (710) among the electronic equipment may be one or more, and in FIG. 19, one processor (710) is given as an example; the memory (720) is configured to store one or more programs; and when the one or more programs are executed by the one or more processors (710), the one or more processors (710) implement a loop filtering method, a video encoding method, or a video decoding method according to an embodiment of the present application.
[0408] The electronic equipment further includes a communication device (730), an input device (740), and an output device (750).
[0409] The processor (710), memory (720), communication device (730), input device (740), and output device (750) among the electronic equipment may be connected by a bus or other method, and in FIG. 19, a connection via a bus is given as an example.
[0410] The input device (740) is configured to receive input numeric or character information and to generate key signal inputs related to user settings and function control of the electronic equipment. The output device (750) may include display equipment such as a display screen.
[0411] The communication device (730) may include a receiver and a transmitter. The communication device (730) is configured to perform information transmission and reception communication under the control of the processor (710).
[0412] The memory (720) is a computer-readable storage medium and may be configured to store software programs, computer-executable programs, and modules, for example, may be configured to store program instructions / modules corresponding to the loop filtering method according to an embodiment of the present application (e.g., a video unit acquisition module (410), a classification module (420), a determination module (430), and a control module (440) in the loop filtering device). The memory (720) may include a program storage area and a data storage area, wherein the program storage area may store an application program required for an operating system or at least one function; and the data storage area may store data generated according to the use of electronic equipment. Additionally, the memory (720) may include a high-speed random access memory and may also include a non-volatile memory such as at least one magnetic disk storage element, a flash storage element, or other non-volatile solid-state storage element. In some examples, the memory (720) may additionally include memory installed remotely to the processor (710), and such remote memory may be connected to electronic equipment via a network. Examples of the network include, but are not limited to, the Internet, intranet, local area network, mobile network, and combinations thereof.
[0413] Embodiments of the present application further provide a storage medium, said storage medium has a computer program stored therein, and when said computer program is executed by a processor, a method according to any one of the embodiments of the present application is implemented. The method may be a loop filtering method, said loop filtering method comprising: a step of obtaining a first reconstructed video unit processed by a loop filter; a step of performing pixel classification on said first reconstructed video unit; a step of determining a scale factor corresponding to each class; and a step of obtaining a second reconstructed video unit by performing scale adjustment on said first reconstructed video unit according to said scale factor.
[0414] Alternatively, the method may be a video encoding method, wherein the video encoding method comprises: a step of obtaining a first reconstructed picture by performing filtering on a reconstructed picture of a current picture; a step of obtaining a second reconstructed picture by performing filtering processing on the first reconstructed picture using a loop filtering method; and a step of identifying loop filtering control information of the loop filtering method in a bitstream based on the filtering processing; wherein the first reconstructed picture includes a first reconstructed video unit and the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined according to the loop filtering method of any embodiment.
[0415] Alternatively, the method may be a video decoding method, wherein the video decoding method comprises the steps of: obtaining a bitstream containing an encoded video sequence and obtaining loop filtering control information related to a loop filtering method; performing filtering on a reconstructed picture of a current picture to obtain a first reconstructed picture; and performing filtering processing on the first reconstructed picture using a loop filtering method based on the loop filtering control information to obtain a second reconstructed picture; wherein the first reconstructed picture includes a first reconstructed video unit and the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined according to the loop filtering method of any embodiment.
[0416] The computer storage medium of the embodiments of the present application may use any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, a semiconductor system, device, or element, or any combination thereof, but is not limited thereto. More specific examples (non-consumable list) of computer-readable storage media include electrical connections having one or more wires, portable computer magnetic disks, hard disks, Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM), flash memory, portable CD-ROM, optical storage elements, magnetic storage elements, or any suitable combination thereof. The computer-readable storage medium may be any type of medium that contains or stores a program, and said program may be used by an instruction execution system, device, or element, or by a combination thereof.
[0417] A computer-readable signal medium may include a data signal that is propagated in the baseband or as part of a carrier wave, the data signal of which includes computer-readable program code. The propagated data signal may take various forms and may include, but is not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and such computer-readable medium may transmit, propagate, or transmit a program that is used by an instruction execution system, device, or element, or by a combination thereof.
[0418] Program code contained in a computer-readable medium may be transmitted using any suitable medium, which includes, but is not limited to, wireless, wire, optical cable, radio frequency (RF), etc.
[0419] Computer program code for performing the operation of the present application may be written in one or more programming languages or a combination thereof, said programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, and further include existing procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on a user computer, partially on a user computer, as a single independent software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer may be connected to the user computer or to an external computer (e.g., connected via the Internet using an Internet service provider) through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN).
[0420] The above description is merely an exemplary embodiment of the present application, and the scope of protection of the present application is not limited thereto.
[0421] Persons skilled in the art should understand that the term "user terminal" includes any suitable type of wireless user equipment, such as mobile phones, portable data processing devices, portable web browsers, or vehicle-mounted mobile stations.
[0422] Generally, various embodiments of the present application may be implemented in hardware or special purpose circuitry, software, logic, or any combination thereof. For example, in some aspects, it may be implemented in hardware, and in other aspects, in firmware or software that can be executed by a controller, microprocessor, or other computing device, but the present application is not limited thereto.
[0423] Embodiments of the present application may be implemented by a data processor of a mobile device executing computer program instructions, for example, by a processor entity, by hardware, or by a combination of software and hardware. Computer program instructions may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.
[0424] Any block diagram of a logic flow in the drawings of the present application may indicate program steps, may indicate interconnected logic circuits, modules, and functions, or may indicate a combination of program steps and logic circuits, modules, and functions. A computer program may be stored in memory. Memory may be of any type suitable for the local technology environment and may be realized by any suitable data storage technology, such as, for example, Read-Only Memory (ROM), Random Access Memory (RAM), optical memory devices and systems (such as Digital Video Disc (DVD) or Compact Disc (CD), but not limited thereto). Computer-readable media may include non-transient storage media. Data processors may be of any type suitable for the local technology environment and may be, for example, general-purpose computers, dedicated computers, microprocessors, Digital Signal Processing (DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FGPA), and processors based on multi-core processor architectures, but not limited thereto.
Claims
Claim 1 A loop filtering method comprising: a step of obtaining a first reconstructed video unit processed by a loop filter; a step of performing pixel classification on the first reconstructed video unit; a step of determining a scale factor corresponding to each class; and a step of performing scale adjustment on the first reconstructed video unit according to the scale factor to obtain a second reconstructed video unit; wherein the step of determining a scale factor corresponding to each class comprises: a step of integrating pixel points of the same class into the same coordinate diagram - wherein the horizontal coordinate of each pixel point is the output of each pixel point after passing through the loop filter, and the vertical coordinate of each pixel point is the input of each pixel point before passing through the loop filter -; a step of obtaining a straight line by performing fitting on the pixel points in the coordinate diagram based on the least squares method and using the slope of the straight line as a scale factor corresponding to the corresponding class; or a step of reading loop filtering control information related to a loop filtering method to determine a scale factor corresponding to each class. Claim 2 A loop filtering method according to claim 1, wherein the loop filter comprises a neural network-based loop filter; or wherein the first reconstructed video unit and the second reconstructed video unit each comprise one of a reconstructed picture, a slice, a code block, or a code tree block. Claim 3 In claim 1, the step of performing scale adjustment for the first reconstructed video unit according to the scale factor comprises at least one of: the step of performing scale adjustment for the luminance component of the first reconstructed video unit according to the scale factor; the step of performing scale adjustment for the first chroma component of the first reconstructed video unit according to the scale factor; and the step of performing scale adjustment for the second chroma component of the first reconstructed video unit according to the scale factor; or, wherein the step of performing scale adjustment for the first reconstructed video unit according to the scale factor comprises, for each pixel point of each class, calculating the loss between the pixel after performing scale adjustment using the scale factor corresponding to each class and the original pixel; and, according to the loss corresponding to the pixel points of a plurality of classes, performing scale adjustment for the pixel points of the corresponding class using the scale factor corresponding to some of the plurality of classes; or calculating the first loss between the picture and the original picture after performing scale adjustment for the pixel points of a plurality of classes using a single scale factor; Calculating a second loss between the picture and the original picture after performing scaling on pixel points of a corresponding class using scale factors corresponding to a plurality of classes respectively; and, depending on the first loss and the second loss, performing scaling on the first reconstructed video unit using a single scale factor or performing scaling on pixel points of a corresponding class using scale factors corresponding to a plurality of classes respectively—wherein the single scale factor is a value set according to statistical information; and the single scale factor is determined according to the slope of a straight line obtained by fitting each pixel point in the first reconstructed video unit—;Alternatively, calculating a second loss between a video unit and a raw video unit after performing scaling on pixel points of a corresponding class using scale factors corresponding to a plurality of classes respectively; fusing the classes of difference values of the corresponding scale factors within a set range into one class; determining a scale factor corresponding to each fused class; calculating a third loss between a video unit and a raw video unit after performing scaling on pixel points of a corresponding fused class using scale factors corresponding to a plurality of fused classes respectively; and, depending on the second loss and the third loss, performing scaling on pixel points of a corresponding class using scale factors corresponding to the plurality of classes respectively, or performing scaling on pixel points of a corresponding fused class using scale factors corresponding to the plurality of fused classes respectively; or, for each pixel point of a class, calculating a loss between a pixel and a raw pixel after performing scaling using scale factors corresponding to each class; A step of identifying, respectively, the switch state of scale adjustment corresponding to the plurality of classes using a scale adjustment switch flag according to the loss corresponding to the pixel points of the plurality of classes; and, according to the scale adjustment switch flag, performing scale adjustment on the pixel points of the corresponding class using a scale factor corresponding to the class for which scale adjustment is turned on; or calculating a first loss between the video unit and the raw video unit after performing scale adjustment on the pixel points of the plurality of classes using a single scale factor; and calculating a second loss between the video unit and the raw video unit after performing scale adjustment on the pixel points of the corresponding class using a scale factor corresponding to the plurality of classes.A loop filtering method comprising one of the steps of: identifying, respectively, a switch state of scale control corresponding to a single scale factor and a switch state of scale control corresponding to a plurality of classes using a scale control switch flag according to the first loss and the second loss; and, according to the scale control switch flag, performing scale control on pixel points of the plurality of classes using the single scale factor or performing scale control on pixel points of a corresponding class using a scale factor corresponding to the class for which scale control is turned on. Claim 4 In claim 1, the pixel classification method comprises: performing pixel classification based on a luminance component; performing pixel classification based on a first chroma component; performing pixel classification based on a second chroma component; performing pixel classification based on a block classification mode; performing pixel classification based on a pixel point region; performing pixel classification based on a class of a neural network-based loop filter; performing pixel classification based on a classification mode used by a deblocking filter (DBF); performing pixel classification based on a classification mode used by sample adaptive offset (SAO) filtering; and performing pixel classification through a neural network used for pixel classification. A loop filtering method comprising at least one of the following; and a step of performing pixel classification based on the block classification mode, wherein the step of acquiring a pixel block of the first reconstructed video unit; a step of calculating a Laplacian gradient of each pixel block and calculating a directional factor and an activity factor according to the Laplacian gradient; a step of determining the class of the pixel block according to at least one of the directional factor and the activity factor and determining the class of all pixel points of the first reconstructed video unit. Claim 5 A video encoding method comprising: a step of obtaining a first reconstructed picture by performing filtering on a reconstructed picture of a current picture; a step of obtaining a second reconstructed picture by performing filtering processing on the first reconstructed picture using a loop filtering method; and a step of identifying loop filtering control information of the loop filtering method in a bitstream based on the filtering processing, wherein the first reconstructed picture includes a first reconstructed video unit and the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined according to the loop filtering method according to claim 1. Claim 6 A video encoding method according to claim 5, further comprising: a step of performing block classification on the second reconstructed picture after obtaining the second reconstructed picture; a step of performing adaptive loop filtering on pixel points of each class in the second reconstructed picture; or a step of performing adaptive loop filtering on pixel points of each class in the second reconstructed picture according to the classification result of the pixel classification. Claim 7 A video encoding method according to claim 5, further comprising: a step of performing block classification on the first reconstructed picture before performing filtering processing on the first reconstructed picture using the loop filtering method; a step of performing adaptive loop filtering on pixel points of each class among the first reconstructed picture according to the classification result of the block classification; or a step of performing deblocking filter (DBF) filtering on the reconstructed picture; a step of performing sample adaptive offset (SAO) filtering on the second reconstructed picture after obtaining the second reconstructed picture; or a step of performing DBF filtering and SAO filtering on the reconstructed picture. Claim 8 A video encoding method according to claim 5, further comprising the step of performing adaptive loop filtering on pixel points of each class in the first reconstructed picture according to the classification result of pixel classification, before determining the scale factor corresponding to each class. Claim 9 A video decoding method comprising: a bitstream including an encoded video sequence and obtaining loop filtering control information related to a loop filtering method; a step of performing filtering on a reconstructed picture of a current picture to obtain a first reconstructed picture; and a step of performing filtering processing on the first reconstructed picture using a loop filtering method based on the loop filtering control information to obtain a second reconstructed picture, wherein the first reconstructed picture includes a first reconstructed video unit and the second reconstructed picture includes a second reconstructed video unit, and the loop filtering method is determined according to the loop filtering method of claim 1. Claim 10 A video decoding method according to claim 9, wherein the loop filtering control information comprises at least one of: a sequence-level filtering switch flag of the encoded video sequence used to instruct the loop filtering method to be turned on or off for the video sequence in which the current picture is located; a filtering picture-level flag used to instruct the loop filtering control information to appear in picture header information or slice header information; and loop filter structure information used to instruct the loop filtering to be performed for a video unit to which loop filtering is applied. Claim 11 In claim 10, the video unit is a video decoding method comprising a picture or video slice in the video. Claim 12 In claim 10, the loop filter structure information comprises at least one of: a video unit filtering switch flag used to instruct a video unit to turn on or off the loop filtering method; a video unit component switch flag used to indicate a video unit component to which loop filtering is applied; a video unit class count used to indicate the total number of classes of the video unit pixel classification; a video unit class filtering switch flag used to instruct a plurality of classes among the video units to turn on or off the loop filtering method; and a video unit scale factor used to indicate a scale factor corresponding to the video unit or a class among the video units; wherein, the video unit component comprises: a luminance component of the video unit; a first chroma component of the video unit; and a second chroma component of the video unit. A video decoding method comprising at least one of the following: the step of performing filtering processing on the first reconstructed picture using a loop filtering method based on the loop filtering control information, wherein the step of identifying a video unit in the first reconstructed picture to which the loop filtering method is turned on according to the video unit filtering switch flag; the step of identifying a video unit component to which loop filtering is applied and a class in which the loop filtering method is turned on in the video unit according to the video unit component switch flag and the video unit class filtering switch flag; and the step of performing scale adjustment on the video unit component of the class to which the loop filtering method is turned on using the loop filtering method according to the video unit scale factor. Claim 13 A video decoding method according to claim 9, further comprising the step of performing adaptive loop filtering on pixel points of each class among the second reconstructed picture according to the classification result of pixel classification after obtaining the second reconstructed picture. Claim 14 A video decoding method according to claim 9, further comprising: a step of performing block classification on the first reconstructed video unit before performing filtering processing on the first reconstructed picture using a loop filtering method based on the loop filtering control information; and a step of performing adaptive loop filtering on pixels of each class among the first reconstructed picture according to the classification result of the block classification. Claim 15 A video decoding method according to claim 9, further comprising the step of performing adaptive loop filtering on pixels of each class among the first reconstructed video units according to the classification result of pixel classification, before determining the scale factor corresponding to each class. Claim 16 A video decoding method according to claim 9, further comprising the step of performing deblocking filter (DBF) filtering on the reconstructed picture before obtaining a second reconstructed picture by performing filtering processing on the first reconstructed picture using a loop filtering method based on the loop filtering control information; and further comprising the step of performing sample adaptive offset (SAO) filtering on the decoded reconstructed picture after obtaining the second reconstructed picture. Claim 17 A video decoding method according to claim 9, further comprising the step of performing DBF filtering and SAO filtering on the reconstructed picture before performing filtering processing on the first reconstructed picture using a loop filtering method based on the loop filtering control information. Claim 18 An electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor; wherein, when the processor executes the program, the device implements a loop filtering method according to any one of claims 1 to 4, a video encoding method according to any one of claims 5 to 8, or a video decoding method according to any one of claims 9 to 17. Claim 19 A non-volatile computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by a processor, a loop filtering method according to any one of claims 1 to 4, a video encoding method according to any one of claims 5 to 8, or a video decoding method according to any one of claims 9 to 17 is implemented. Claim 20 delete Claim 21 delete Claim 22 delete Claim 23 delete Claim 24 delete Claim 25 delete Claim 26 delete Claim 27 delete Claim 28 delete Claim 29 delete Claim 30 delete Claim 31 delete Claim 32 delete Claim 33 delete Claim 34 delete
Citation Information
Patent Citations
Image filtering apparatus, image decoding apparatus, and image coding apparatus
US20210021820A1
