Loop filtering method and related apparatus
By obtaining the target classification category and reference pixel vector in the video frame, and combining local and non-local reference pixels for filtering, the problem of insufficient adaptability in the existing technology is solved, and the video quality and adaptability are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PENG CHENG LAB
- Filing Date
- 2022-11-15
- Publication Date
- 2026-07-03
Smart Images
Figure CN115941977B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video coding technology, and in particular to a loop filtering method and related apparatus. Background Technology
[0002] Loop filtering technology has attracted widespread attention due to its significant noise reduction capabilities in hybrid video coding frameworks. However, the loop filters in the H.266 / VVC standard generally employ methods based on spatial local priors. Representative works include Deblocking Filter (DBF), Sample Adaptive Offset (SAO), Adaptive Loop Filter (ALF), and Constrained Directional Enhancement Filter (CDEF). These methods utilize the pixel to be filtered and its spatial neighbors for filtering operations, and are characterized by relatively low complexity and ease of hardware and software implementation. However, all of the above methods are limited by the coding rules and are mainly designed based on the spatial local prior characteristics of the image, resulting in insufficient adaptability of the loop filters. They are difficult to adapt to complex video content, thus affecting the subjective and objective quality of compressed video.
[0003] Therefore, the existing technology still needs to be improved and enhanced. Summary of the Invention
[0004] The technical problem to be solved by this application is to provide a loop filtering method and related apparatus to address the shortcomings of the prior art.
[0005] To address the aforementioned technical problems, the first aspect of this application provides a loop filtering method, the method comprising:
[0006] For each pixel in the video frame to be filtered, the target classification category to which the pixel belongs is obtained, and the reference pixel vector of the pixel is obtained, wherein the reference pixel vector includes local reference pixels and non-local reference pixels;
[0007] The pixels to be filtered are filtered based on the reference pixel vector and the target classification category to obtain the filtered video frame.
[0008] In one implementation, the classification method used for the pixels to be filtered is a content-aware classification method; obtaining the target classification category to which the pixels to be filtered belong specifically includes:
[0009] The pixel to be filtered is classified by a block-level classifier to obtain a classification category based on block-level features;
[0010] The pixel to be filtered is classified by a pixel-level classifier to obtain a classification category based on pixel-level features;
[0011] Based on the block-level feature-based classification category and the pixel-level feature-based classification category, the target classification category of the pixel to be filtered is determined.
[0012] In one implementation, obtaining the reference pixel vector of the pixel to be filtered specifically includes:
[0013] The resolution of the video frame to be filtered is obtained, and the first quantity and the second quantity corresponding to the video frame to be filtered are determined according to the resolution and the classification category based on block-level features, wherein the first quantity is the number of local parameter pixels and the second quantity is the number of non-local parameter pixels.
[0014] Obtain a first number of local reference pixels and a second number of non-local reference pixels corresponding to the pixel to be filtered;
[0015] The obtained local reference pixels and non-local reference pixels are concatenated to obtain the reference pixel vector.
[0016] In one implementation, obtaining the first number of local reference pixels and the second number of non-local reference pixels corresponding to the pixel to be filtered specifically includes:
[0017] Based on the pixel distance between pixels, a first number of local reference pixels are obtained in the video frame to be filtered;
[0018] A number of search video frames corresponding to the video frame to be filtered are obtained, and a second number of non-local reference pixels are obtained from the number of search video frames by block matching, wherein the search video frames include at least the video frame to be filtered.
[0019] In one implementation, obtaining the plurality of search video frames corresponding to the video frame to be filtered specifically includes:
[0020] Obtain the frame type of the video frame to be filtered;
[0021] When the frame type is I-frame, the video frame to be filtered is used as the search video frame;
[0022] When the frame type is B-frame or P-frame, a reference frame is selected for the video frame to be filtered, and the selected reference frame and the video frame to be filtered are used as the search video frame.
[0023] In one implementation, determining the first and second quantities of the video frames to be filtered based on the resolution and the block-level feature-based classification category specifically includes:
[0024] Obtain the resolution category to which the resolution belongs;
[0025] Based on the resolution category and the block-level feature-based classification category, a first quantity corresponding to the video frame to be filtered is selected from a preset first quantity list.
[0026] The second quantity is determined based on the preset filtering quantity and the first quantity.
[0027] In one implementation, the step of filtering the pixel to be filtered based on the reference pixel vector and the target classification category to obtain the filtered video frame specifically includes:
[0028] The filtering coefficients corresponding to the pixels to be filtered are determined based on the target classification category.
[0029] The filtered pixel corresponding to the pixel to be filtered is determined based on the filtering coefficients and the reference pixel vector.
[0030] The image formed by the filtered pixels corresponding to each pixel to be filtered is taken as the filtered video frame corresponding to the video frame to be filtered.
[0031] A second aspect of this application provides a loop filtering system, the system comprising:
[0032] The acquisition module is used to acquire, for each pixel to be filtered in the video frame to be filtered, the target classification category to which the pixel to be filtered belongs, and the reference pixel vector of the pixel to be filtered, wherein the reference pixel vector includes local reference pixels and non-local reference pixels;
[0033] The filtering module is used to filter the pixel to be filtered according to the reference pixel vector and the target classification category to obtain the filtered video frame.
[0034] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in any of the loop filtering methods described above.
[0035] A fourth aspect of this application provides a terminal device, which includes: a processor, a memory, and a communication bus; the memory stores a computer-readable program that can be executed by the processor;
[0036] The communication bus enables communication between the processor and the memory;
[0037] When the processor executes the computer-readable program, it implements the steps in any of the loop filtering methods described above.
[0038] Beneficial Effects: Compared with existing technologies, this application provides a loop filtering method and related apparatus. The method includes, for each pixel in a video frame to be filtered, obtaining the target classification category to which the pixel belongs, and obtaining a reference pixel vector for the pixel, wherein the reference pixel vector includes local reference pixels and non-local reference pixels; filtering the pixel based on the reference pixel vector and the target classification category to obtain a filtered video frame. This application fuses the local and non-local reference pixels of the pixel to be filtered to obtain a reference pixel vector, which can effectively improve the subjective and objective quality of compressed video. Furthermore, by determining the number of local and non-local reference pixels based on the resolution of the video frame to be filtered, the reference pixel vector can be adaptively selected according to the content complexity of the video frame, improving the adaptability of the loop filtering. Attached Figure Description
[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1 A flowchart of the loop filtering method provided in this application.
[0041] Figure 2 A flowchart illustrating the loop filtering method provided in this application.
[0042] Figure 3 This is a schematic diagram illustrating the process of acquiring local reference pixels and non-local reference pixels.
[0043] Figure 4 The schematic diagram of the loop filter system provided in this application.
[0044] Figure 5 A schematic diagram of the terminal device provided in this application. Detailed Implementation
[0045] This application provides a loop filtering method and related apparatus. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.
[0046] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.
[0047] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0048] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.
[0049] The inventors discovered that loop filtering technology has garnered widespread attention due to its significant noise reduction capabilities in hybrid video coding frameworks. However, loop filters in the H.266 / VVC standard commonly employ methods based on spatial local priors. Representative works include Deblocking Filter (DBF), Sample Adaptive Offset (SAO), Adaptive Loop Filter (ALF), and Constrained Directional Enhancement Filter (CDEF). These methods utilize the pixel to be filtered and its spatial neighbors for filtering operations, exhibiting relatively low complexity and ease of hardware and software implementation. However, all of these methods are constrained by coding rules and are primarily designed based on local spatial prior characteristics of the image, resulting in insufficient adaptability of the loop filters. This makes it difficult to adapt to complex video content, thus affecting the subjective and objective quality of compressed video. The following section provides a detailed explanation of these loop filters.
[0050] The DBF loop filter uses a series of low-pass filters at the boundaries of a 4x4 block. These low-pass filters are derived from the encoded information of the reconstructed blocks on both sides of the boundary, such as prediction modes and motion vectors. Although DBF can significantly improve the subjective quality of the reconstructed video, it only applies weighted processing at the boundaries of the CU (Coding Unit: the basic unit that performs prediction, transform, quantization, and entropy coding), PU (Prediction Unit: the basic unit for intra-frame and inter-frame prediction), and TU (Transform Unit: the basic unit for transform and quantization). Pixels inside the block will still be affected by quantization noise, thus failing to guarantee the subjective quality of the reconstructed video.
[0051] To address the noise issue within pixels within a block, H.265 / HEVC introduces SAO (Side Offset Anomaly) to further reduce compression distortion after DBF (Depth-of-Flatness Compression). SAO obtains offset values by minimizing the average sample distortion between different regions of the reconstructed image after DBF and the original image. Furthermore, it corrects the reconstructed samples after DBF by conditionally adding corresponding offset values to each sample point. SAO includes Edge Offset (EO) and Band Offset (BO) modes. EO mode classifies pixels based on their nearest neighbors, while BO mode classifies pixels based on their pixel values. Pixels of the same category use the same offset value, which is written into the bitstream. Although SAO effectively removes salt-and-pepper noise, its processing capability is limited by the EO and BO classification methods, and its pixel processing approach is relatively simple, thus limiting its encoding performance.
[0052] To further improve the subjective and objective quality of compressed video, H.266 / VVC adopted ALF, a Wiener adaptive filter that uses a weighted average of spatially nearest neighbors to filter the current point. The filter coefficients are trained at the encoder based on minimizing the mean square error between the reconstructed frame and the original frame, and then written into the bitstream for transmission to the decoder. To obtain a highly adaptive and low-complexity adaptive loop filter, Geometry Transformation-based Adaptive Loop Filter (GALF) was proposed and ultimately adopted by the H.266 / VVC standard. GALF uses gradients as the basis for pixel classification and, without changing the filter shape, sets several different usage forms for each set of filter coefficients, including rotation, diagonal flipping, and vertical flipping, effectively improving coding performance. In subsequent JVET meetings, the Non-linear Adaptive Loop Filter (NALF) and Cross Component Adaptive Loop Filter (CCALF) techniques were also adopted. NALF, CCALF, and NALF together form the ALF module in the current H.266 / VVC standard. ALF is the most gain-significant filtering tool in H.266 / VVC; however, it also consumes significant hardware and software resources.
[0053] While some loop filters based on nonlocal priors can compensate for this drawback, the unsupervised parameter estimation methods widely used in nonlocal filters limit their compression performance. Furthermore, the filtering method of transform domain coefficient decomposition in nonlocal filtering also leads to high encoding and decoding complexity. Loop filters based on spatial nonlocal priors mainly utilize the nonlocal self-similarity of images and are often called non-local filters in video coding. Non-local filtering in video coding originated from the classic non-local means filter (NLM) in image processing, which uses a weighted average of nonlocal similar blocks to reduce noise. The weights are determined by the similarity between small blocks located at the filter location and the reference block location. In the field of video coding, NLM was first proposed as a loop filter in H.265 / HEVC. This method can compensate for the shortcomings of existing loop filters based on local smoothing prior models of images; however, it fails to fully consider the characteristics of compressed signals, resulting in low compression performance. Subsequently, Han et al. proposed a quadtree-based nonlocal filter to improve the filter's flexibility and content adaptability. Next, Zhang et al. proposed a Nonlocal Structure-based Loop Filter (NLSF). This method reduces distortion by performing Singular Value Decomposition (SVD) on similar structure groups and then applying hard thresholding to shrink the decomposed singular values. To improve the filter's adaptability to noise intensity, Zhang et al. designed a Nonlocal Adaptive Loop Filter using a low-rank image prior model. Content-based filtering threshold parameter settings in NALF significantly improve filter performance, and the combination of frame-level and CTU-level control switches increases filter flexibility. To reduce the complexity of the NLSF block matching process, a heuristic search strategy was proposed, which can reduce decoding time by approximately 70% with almost no impact on performance. To further reduce algorithm complexity, a multi-granularity parallel optimization method for nonlocal filtering on heterogeneous platforms was proposed. However, the similar block search and SVD processes still result in excessively high encoding and decoding complexity. Furthermore, the floating-point SVD process also presents challenges for hardware implementation. Therefore, the practicality and standardization of nonlocal loop filtering is an urgent problem to be solved.
[0054] To address the aforementioned issues, in this embodiment, for each pixel in the video frame to be filtered, the target classification category to which the pixel belongs and a reference pixel vector are obtained. The reference pixel vector includes local reference pixels and non-local reference pixels. The pixel is then filtered based on its reference pixel vector and target classification category to obtain a filtered video frame. This application fuses the local and non-local reference pixels of the pixel to be filtered to obtain a reference pixel vector, which effectively improves the subjective and objective quality of the compressed video. Furthermore, by determining the number of local and non-local reference pixels based on the resolution of the video frame to be filtered, the reference pixel vector can be adaptively selected according to the content complexity of the video frame, improving the adaptability of the loop filtering.
[0055] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.
[0056] This embodiment provides a loop filtering method, such as Figure 1 and 2 As shown, the method includes:
[0057] S10. For each pixel in the video frame to be filtered, obtain the target classification category to which the pixel belongs, and obtain the reference pixel vector of the pixel.
[0058] Specifically, the video frame to be filtered is a compressed video frame, which may contain image noise. The target classification category is the category of the pixel to be filtered. The target classification category is determined based on a content-aware classification method, or it can be a classification method that directly uses pixel value size. The content-aware classification method combines block-level features and pixel-level features. Block-level features are used to reflect the strong correlation between similarity and noise level, while pixel-level features are used to determine the correlation between pixels in the video frame to be filtered.
[0059] In one implementation, the classification method used for the pixels to be filtered is a content-aware classification method, which matches the classification method with the subsequent acquisition methods of local reference pixels and non-local reference pixels, thereby improving the filtering performance of the loop filter. Based on this, obtaining the target classification category to which the pixel to be filtered belongs specifically includes:
[0060] The pixel to be filtered is classified by a block-level classifier to obtain a classification category based on block-level features;
[0061] The pixel to be filtered is classified by a pixel-level classifier to obtain a classification category based on pixel-level features;
[0062] Based on the block-level feature-based classification category and the pixel-level feature-based classification category, the target classification category of the pixel to be filtered is determined.
[0063] Specifically, the block-level classifier classifies video blocks based on their similarity to obtain the block-level feature-based classification category of the pixel to be filtered. The block-level classifier can be represented as:
[0064]
[0065] Where K1 represents the number of categories in the block-level classifier. For the pre-set correlation threshold, This indicates the similarity between video blocks.
[0066] Furthermore, when classifying using a block-level classifier, the video frame to be filtered can be pre-divided into several video blocks. Then, based on the similarity of each video block, the classification category containing the pixel to be filtered can be determined, and this classification category can be used as the block-level feature-based classification category corresponding to the pixel to be filtered.
[0067] The pixel-level classifier is used to classify pixels based on their pixel values in the video to be filtered, so as to obtain the classification category of the pixels to be filtered based on pixel-level features. The pixel-level classifier can be represented as follows:
[0068]
[0069] Where BD represents the bit depth inside the encoder, K2 represents the number of categories in the pixel-level classifier, and z(p,t) represents the pixel to be filtered.
[0070] After obtaining the block-level feature-based classification category and the pixel-level feature-based classification category, the target classification category corresponding to the pixel to be filtered can be determined by taking the square root of the block-level feature-based classification category and the pixel-level feature-based classification category. This allows the target classification category to fully utilize the local and non-local features in the video frame to be filtered. The target classification category can be determined using a Cartesian product, and correspondingly, the target classification category can be represented as:
[0071]
[0072] Where C represents the classification category of the current pixel z(p,t), K represents the number of categories, and K = K1 × K2.
[0073] In one implementation, based on the noise distribution characteristics of the screen content video and considering the relatively simple content and texture characteristics of the screen content video, the target classification adopts a classification method that directly classifies based on pixel value size. This classification method, which directly classifies based on pixel value size, can be expressed as:
[0074]
[0075] Where C represents the classification category of the current pixel z(p,t), X represents the number of categories, for example, X=32, and BD represents the bit depth inside the encoder.
[0076] The reference pixel vector includes local reference pixels and non-local reference pixels. Local reference pixels are determined based on pixel distance, while non-local reference pixels are determined based on pixel blocks. The reference pixel vector is obtained by combining local and non-local reference pixels. In this embodiment, the reference pixel vector combines local spatial correlation, non-local spatial correlation, and temporal correlation, thus improving the subjective and objective quality of the filtered video frame determined based on the filtered pixels.
[0077] The first number of local reference pixels and the second number of non-local reference pixels included in the reference pixel vector can be determined based on the classification method corresponding to the pixel to be filtered. When the classification method corresponding to the pixel to be filtered is a content-aware classification method, the first and second numbers are determined according to the classification category corresponding to the block-level feature and the resolution of the video to be filtered. When the classification method corresponding to the pixel to be filtered is a classification method that directly uses the pixel value size for classification, the first and second numbers can be preset.
[0078] In one implementation, when the classification method used for the pixel to be filtered is a content-aware classification method, obtaining the reference pixel vector of the pixel to be filtered specifically includes:
[0079] Obtain the resolution of the video frame to be filtered, and determine the first number and the second number of the video frame to be filtered based on the resolution and the classification category based on block-level features;
[0080] Obtain a first number of local reference pixels and a second number of non-local reference pixels corresponding to the pixel to be filtered;
[0081] The obtained local reference pixels and non-local reference pixels are concatenated to obtain the reference pixel vector.
[0082] Specifically, the vector dimension of the reference pixel vector is equal to the sum of the first and second quantities. Accumulated, the reference pixel vector includes all acquired local reference pixels and all non-local reference pixels. Furthermore, when concatenating the acquired local and non-local reference pixels, the reference pixel vector can be obtained by concatenating the local and non-local reference pixels in the order of local and non-local reference pixels, or by concatenating the non-local and local reference pixels in the same order.
[0083] In one implementation, obtaining the first number of local reference pixels and the second number of non-local reference pixels corresponding to the pixel to be filtered specifically includes:
[0084] The step of determining the first and second quantities of the video frames to be filtered based on the resolution and the block-level feature-based classification category specifically includes:
[0085] Obtain the resolution category to which the resolution belongs;
[0086] Based on the resolution category and the block-level feature-based classification category, a first quantity corresponding to the video frame to be filtered is selected from a preset first quantity list.
[0087] The second quantity is determined based on the preset filtering quantity and the first quantity.
[0088] Specifically, the preset filtering quantity is a pre-set number of reference pixels required when filtering the pixel to be filtered. The preset filtering quantity is equal to the vector dimension of the reference pixel vector, that is, the preset filtering quantity = first quantity + second quantity.
[0089] The resolution category is determined based on the resolution size. For example, the resolution categories include a first category, a second category, and a third category. The first category consists of resolutions greater than 1920*1080, the second category consists of resolutions less than 1280*70, and the third category consists of all resolutions other than the first and second categories. The preset first quantity list is determined based on the resolution category and the classification category based on block-level features. After obtaining the resolution category and the classification category based on block-level features, a first quantity can be selected from the preset first quantity list. For example, the first quantity list is the list shown in Table 1.
[0090] Table 1 First Quantity List
[0091]
[0092] This embodiment selects a first number of local reference pixels for each pixel to be filtered based on its block-level feature-based classification and the resolution of the video frame. This adaptively adjusts the number of local reference pixels and non-local reference pixels according to the complexity of the video block in which the pixel is located, improving the adaptability of the loop filtering. For example, more local reference pixels are used for low-complexity and high-complexity video blocks, while more non-local reference pixels are used for medium-complexity blocks. Of course, in practical applications, the first number list can be set according to the actual usage; this embodiment only provides a typical implementation.
[0093] In one implementation, such as Figure 3 As shown, obtaining the first number of local reference pixels and the second number of non-local reference pixels corresponding to the pixel to be filtered specifically includes:
[0094] Based on the pixel distance between pixels, a first number of local reference pixels are obtained in the video frame to be filtered;
[0095] A number of search video frames corresponding to the video frame to be filtered are obtained, and a second number of non-local reference pixels are obtained from the number of search video frames by block matching, wherein the search video frames include at least the video frame to be filtered.
[0096] Specifically, local reference pixels are determined based on pixel distances within the video frame to be filtered. In other words, for each pixel to be filtered, the pixel distances between that pixel and all other pixels can be obtained. Then, local reference pixels are selected based on these pixel distances. When selecting local reference pixels based on pixel distances, they can be obtained in order of proximity to the current pixel to be filtered, from closest to furthest, to obtain the local reference pixels. Where, r i , where represents the i-th local parameter pixel, and L represents the first quantity.
[0097] Non-local reference pixels are obtained by block matching within a preset search area. Therefore, when determining non-local reference pixels, the video frame z(t) to be filtered can be divided into B... s ×B s The video blocks are divided into blocks of varying sizes. Following a raster scan order, one block is extracted every m pixels, where m represents the horizontal or vertical distance between two adjacent blocks, indicating the degree of overlap. For each video block, R is obtained within a preset search area. N There are 10 similar blocks, where the selection rule for similar blocks is the L2 norm between image blocks. The L2 norm is calculated as follows:
[0098]
[0099] Where z(p, t) represents the current video block, (p, t) represents the spatial coordinates of the top-left pixel of the current video block in the video frame z(t) to be filtered, z(p′, t′) is a candidate reference block in the preset search region, and p′∈Ω p .
[0100] Furthermore, when filtering compressed video, the compressed video includes I-frames, B-frames, and P-frames, and each I-frame, B-frame, and P-frame carries different amounts of image content. Therefore, when obtaining non-local reference pixels, it is necessary to determine the search video frame corresponding to the video to be filtered based on the frame type of the video frame to be filtered, and then select non-local parameter pixels from the search video frame so that the non-local parameter pixels include both spatial and temporal features. Based on this, in one implementation, obtaining several search video frames corresponding to the video frame to be filtered specifically includes:
[0101] Obtain the frame type of the video frame to be filtered; when the frame type is I-frame, use the video frame to be filtered as the search video frame;
[0102] When the frame type is B-frame or P-frame, a reference frame is selected for the video frame to be filtered, and the selected reference frame and the video frame to be filtered are used as the search video frame.
[0103] Specifically, for an I-frame, similar blocks are searched within the video frame to be filtered. For B-frames and P-frames, similar blocks can be searched within a search video frame consisting of the video frame to be filtered and a reference frame. Correspondingly, the search video frame can be represented as:
[0104]
[0105] Among them, Ω t This represents the video frame being searched, where t represents the video frame to be filtered. i N represents the video frame with parameters i. ref This indicates the number of reference video frames. The reference frames can be the preceding video frames of the video frame to be filtered, or the I-frames corresponding to the video frame to be filtered, etc.
[0106] Furthermore, when obtaining a second number of non-local reference pixels in several search video frames using block matching, a preset search region corresponding to each search video frame can be predetermined. For the video frame to be filtered, the preset search region can be a W1×W1 search window centered on the video block where the pixel to be filtered is located. The search window in the reference frame can be a W2×W2 search window centered on the corresponding video block in the reference frame where the video block where the pixel to be filtered is located is located. The W1×W1 and W2×W2 search windows are respectively represented as follows:
[0107]
[0108]
[0109] After obtaining the search window corresponding to each search video frame, based on the principle of minimizing the L2 norm, similar video blocks corresponding to the video block to which the pixel to be filtered belongs are searched in each search window to obtain a similarity block group G. Then, the similar blocks in the similarity block group are column vectorized to obtain the similarity matrix:
[0110]
[0111] Among them, P G for The matrix, R N This represents the number of similar blocks.
[0112] S20. Filter the pixels to be filtered according to the reference pixel vector and target classification category to obtain the filtered video frame.
[0113] Specifically, after obtaining the reference pixel vector and the target classification category, the filtering coefficients corresponding to the pixel to be filtered can be determined based on the target classification category. Then, the pixel to be filtered is determined based on the filtering coefficients and the reference pixel vector to obtain the filtered pixel.
[0114] In one implementation, the step of filtering the pixel to be filtered based on the reference pixel vector and the target classification category to obtain the filtered video frame specifically includes:
[0115] The filtering coefficients corresponding to the pixels to be filtered are determined based on the target classification category.
[0116] The filtered pixel corresponding to the pixel to be filtered is determined based on the filtering coefficients and the reference pixel vector.
[0117] The image formed by the filtered pixels corresponding to each pixel to be filtered is taken as the filtered video frame corresponding to the video frame to be filtered.
[0118] Specifically, determining the filtered pixel corresponding to the pixel to be filtered based on the filtering coefficients and the reference pixel vector can be expressed as follows:
[0119]
[0120] Where f(p,t) represents the filtered pixel, w j R represents the filter coefficients. j The parameter pixel is represented by N, which represents the number of reference pixels.
[0121] In one implementation, the pixel value of a reference pixel in the reference pixel vector may differ from the pixel value of the pixel to be filtered by more than a preset difference threshold, causing deviations in the filter coefficients and adversely affecting coding performance. Therefore, a constraint operation is introduced on the reference pixel vector to limit the pixel values of abnormal reference pixels within a predetermined range.
[0122] For a constrained reference pixel vector, the filtering operation formula is as follows:
[0123]
[0124] in, The constraint function can be calculated using the following formula:
[0125]
[0126] Where, ξ j It refers to the scope of constraints.
[0127] In one implementation, ξ has four candidate values: 8, 32, 256, and 1023. The encoder can select a constraint range for each filter, and the index of the selected constraint range is transmitted to the decoder. The number of filter taps N is set to 25. To better balance the cost of encoding filter coefficients with distortion, a coefficient sharing scheme is employed. Specifically, two adjacent reference samples share filter coefficients, thus reducing the number of filter coefficients from N to N / 2.
[0128] In summary, this embodiment provides a loop filtering method. The method includes, for each pixel in a video frame to be filtered, obtaining the target classification category to which the pixel belongs, and obtaining a reference pixel vector for the pixel, wherein the reference pixel vector includes local reference pixels and non-local reference pixels; filtering the pixel based on the reference pixel vector and the target classification category to obtain a filtered video frame. This application fuses the local and non-local reference pixels of the pixel to be filtered to obtain a reference pixel vector, which can effectively improve the subjective and objective quality of compressed video. Furthermore, by determining the number of local and non-local reference pixels based on the resolution of the video frame to be filtered, the reference pixel vector can be adaptively selected according to the content complexity of the video frame, improving the adaptability of the loop filtering.
[0129] Based on the above loop filtering method, this embodiment provides a loop filtering system, such as... Figure 4 As shown, the system includes:
[0130] The acquisition module 100 is used to acquire, for each pixel to be filtered in the video frame to be filtered, the target classification category to which the pixel to be filtered belongs, and the reference pixel vector of the pixel to be filtered, wherein the reference pixel vector includes local reference pixels and non-local reference pixels.
[0131] The filtering module 200 is used to filter the pixel to be filtered according to the reference pixel vector and the target classification category of the pixel to be filtered in order to obtain a filtered video frame.
[0132] Based on the above loop filtering method, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the loop filtering method as described in the above embodiment.
[0133] Based on the above loop filtering method, this application also provides a terminal device, such as... Figure 5 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.
[0134] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.
[0135] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.
[0136] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.
[0137] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.
[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A loop filtering method, characterized in that, The method includes: For each pixel in the video frame to be filtered, the target classification category to which the pixel belongs is obtained, and the reference pixel vector of the pixel is obtained, wherein the reference pixel vector includes local reference pixels and non-local reference pixels; The pixel to be filtered is filtered according to the reference pixel vector and the target classification category to obtain the filtered video frame; Specifically, obtaining the reference pixel vector of the pixel to be filtered includes: The resolution of the video frame to be filtered is obtained, and the first quantity and the second quantity corresponding to the video frame to be filtered are determined according to the resolution and the classification category based on block-level features. The first quantity is the number of local parameter pixels, and the second quantity is the number of non-local parameter pixels. The classification category based on block-level features is obtained by classifying the pixels to be filtered through a block-level classifier. Obtain a first number of local reference pixels and a second number of non-local reference pixels corresponding to the pixel to be filtered; The obtained local reference pixels and non-local reference pixels are concatenated to obtain the reference pixel vector.
2. The loop filtering method of claim 1, wherein, The classification method used for the pixels to be filtered is a content-aware classification method; obtaining the target classification category to which the pixels to be filtered belong specifically includes: The pixels to be filtered are classified by a block-level classifier to obtain a classification category based on block-level features; The pixels to be filtered are classified by a pixel-level classifier to obtain a classification category based on pixel-level features; Based on the block-level feature-based classification category and the pixel-level feature-based classification category, the target classification category of the pixel to be filtered is determined.
3. The loop filtering method of claim 1, wherein, The step of obtaining the first number of local reference pixels and the second number of non-local reference pixels corresponding to the pixel to be filtered specifically includes: Based on the pixel distance between pixels, a first number of local reference pixels are obtained in the video frame to be filtered; A number of search video frames corresponding to the video frame to be filtered are obtained, and a second number of non-local reference pixels are obtained from the number of search video frames by block matching, wherein the search video frames include at least the video frame to be filtered.
4. The loop filtering method of claim 3, wherein, The specific steps of obtaining the search video frames corresponding to the video frame to be filtered include: Obtain the frame type of the video frame to be filtered; When the frame type is I-frame, the video frame to be filtered is used as the search video frame; When the frame type is B-frame or P-frame, a reference frame is selected for the video frame to be filtered, and the selected reference frame and the video frame to be filtered are used as the search video frame.
5. The loop filtering method of claim 1, wherein, The step of determining the first and second quantities of the video frames to be filtered based on the resolution and the block-level feature-based classification category specifically includes: Obtain the resolution category to which the resolution belongs; Based on the resolution category and the block-level feature-based classification category, select the first quantity corresponding to the video frame to be filtered from the preset first quantity list; The second quantity is determined based on the preset filtering quantity and the first quantity.
6. The loop filtering method of claim 1, wherein, The step of filtering the pixel to be filtered based on the reference pixel vector and the target classification category to obtain the filtered video frame specifically includes: The filtering coefficients corresponding to the pixels to be filtered are determined based on the target classification category. The filtered pixel corresponding to the pixel to be filtered is determined based on the filtering coefficients and the reference pixel vector. The image formed by the filtered pixels corresponding to each pixel to be filtered is taken as the filtered video frame corresponding to the video frame to be filtered.
7. A loop filtering system, characterized by The system includes: The acquisition module is used to acquire, for each pixel to be filtered in the video frame to be filtered, the target classification category to which the pixel to be filtered belongs, and the reference pixel vector of the pixel to be filtered, wherein the reference pixel vector includes local reference pixels and non-local reference pixels; The filtering module is used to filter the pixel to be filtered according to the reference pixel vector and the target classification category to obtain the filtered video frame. Specifically, obtaining the reference pixel vector of the pixel to be filtered includes: The resolution of the video frame to be filtered is obtained, and the first quantity and the second quantity corresponding to the video frame to be filtered are determined according to the resolution and the classification category based on block-level features. The first quantity is the number of local parameter pixels, and the second quantity is the number of non-local parameter pixels. The classification category based on block-level features is obtained by classifying the pixels to be filtered through a block-level classifier. Obtain a first number of local reference pixels and a second number of non-local reference pixels corresponding to the pixel to be filtered; The obtained local reference pixels and non-local reference pixels are concatenated to obtain the reference pixel vector.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the loop filtering method as described in any one of claims 1-6.
9. A terminal device, characterized in that, include: Processor, memory, and communication bus; the memory stores a computer-readable program that can be executed by the processor; The communication bus enables communication between the processor and the memory; When the processor executes the computer-readable program, it implements the steps of the loop filtering method as described in any one of claims 1-6.