Method and apparatus for adaptive loop filter with virtual bounds and multi-sources for video coding

Abstract

A video coding system utilizing chroma ALF or CCALF with multi-source or high-order taps. According to the method of the present invention, a reconstructed pixel is received, where the reconstructed pixel includes a current block, and the current block includes a luma block and one or more chroma blocks. The filtered chroma output originates from a chroma ALF or CCALF (Cross-Component ALF) of a current chroma sample in one of the one or more chroma blocks, where the chroma ALF comprises one or more multi-source chroma samples or luma samples from a current block in a first coverage area of the chroma ALF, or wherein the CCALF comprises one or more multi-source luminance samples from a luminance block in a second coverage area of the CCALF. A filtered reconstructed first chroma block is provided, wherein the filtered reconstructed first chroma block comprises a filtered chroma output.

Description

[0001] Cross-references

[0002] The present invention is a non-provisional application and claims priority to U.S. Provisional Patent Application No. 63 / 368,511 filed on July 15, 2022. The entirety of the U.S. Provisional Patent Application is incorporated herein by reference. Technical Field

[0003] The present invention relates to a video coding system using an adaptive loop filter (ALF), and more particularly to a chroma ALF or a cross component ALF (CCALF) using multi-source taps. Background Art

[0004] Versatile Video Codec (VVC) is the latest international video coding standard (MPEG) developed by the Joint Video Experts Group (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group. The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology-Coded representation of immersive media-Part 3:Versatile video coding, released in February 2021. VVC was developed by adding more codec tools to improve codec efficiency and handle various types of video sources, including 3D video signals, based on its predecessor HEVC (High Efficiency Video coding).

[0005] Figure 1AAn exemplary adaptive Inter / Intra video coding system including a loop process is described. For intra prediction, prediction data is derived from previously coded video data in the current picture (hereinafter also referred to as a picture). For inter prediction 112, motion estimation (abbreviated as ME) is performed on the encoder side and motion compensation (abbreviated as MC) is performed based on the result of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112 and the selected prediction data is provided to adder 116 to form a prediction error, also referred to as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in the video bit stream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with side information (e.g., motion and decoding modes associated with intra-frame prediction and inter-frame prediction) and other information (e.g., parameters associated with the loop filter applied to the underlying image area). Side information associated with intra-frame prediction 110, inter-frame prediction 112, and loop filter 130 is provided to entropy encoder 122, as shown in FIG. Figure 1A As shown. When using inter-frame prediction mode, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a reference picture buffer 134 and used to predict other frames.

[0006] like Figure 1AAs shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC 128 may be subject to various impairments. Therefore, the loop filter 130 is often applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve the video quality. For example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF) may be used. It may be necessary to merge the loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be merged into the bitstream. Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to the High Efficiency Video Codec (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B As shown, except for the transform 118 and the quantization 120, the decoder can use similar or the same functional blocks as the encoder, because the decoder only needs the inverse quantization 124 and the inverse transform 126. Instead of the entropy encoder 122, the decoder uses the entropy decoder 140 to decode the video bitstream into quantized transform coefficients and required coding information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 on the decoder side does not need to perform a mode search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from the entropy decoder 140 without motion estimation.

[0008] According to VVC, similar to HEVC, the input picture is partitioned into non-overlapping square block areas called CTUs (Codec Tree Units). Each CTU can be divided into one or more smaller Codec Units (CUs). The resulting CU partitions can be square or rectangular. In addition, VVC divides CTUs into Prediction Units (PUs) as units for applying prediction processing, such as inter-frame prediction, intra-frame prediction, etc.

[0009] In the present invention, for emerging video coding development beyond VVC, a loop filter (e.g., adaptive loop filter (ALF) for chroma ALF and / or CCALF (cross-component ALF) or any other loop filter) using multiple sources is disclosed to improve coding efficiency. Summary of the invention

[0010] A video encoding method and apparatus using a chroma adaptive loop filter (chroma ALF) or a cross-component adaptive loop filter (CCALF) is disclosed. According to the method, reconstructed pixels are received, wherein the reconstructed pixels include a current block, and the current block includes a luminance block and one or more chroma blocks. A filtered chroma output is derived for a current chroma sample in one or more chroma blocks from a chroma ALF or a CCALF (cross-component ALF), wherein the chroma ALF includes one or more multi-source chroma samples within a first coverage range of the chroma ALF or a luminance sample in the current block, or the CCALF includes one or more multi-source luminance samples within a second coverage range of the CCALF. A first chroma block reconstructed by filtering is provided, wherein the first chroma block reconstructed by filtering includes the filtered chroma output.

[0011] In one embodiment, for chroma ALF, the one or more multi-source chroma samples include one or more pre-deblocking filter (pre-DBF) and / or pre-sample adaptive offset (pre-SAO) chroma samples in the one or more chroma blocks. In one embodiment, for chroma ALF, the one or more multi-source chroma samples include one or more chroma samples of any pre-ALF type in another block of the one or more chroma blocks.

[0012] In one embodiment, for chroma ALF, the one or more multi-source luma samples include one or more pre-deblocking filtered (pre-DBF) and / or pre-sample adaptive offset (pre-SAO) luma samples in a luma block.

[0013] In one embodiment, for CCALF, the one or more multi-source luma samples include one or more luma output samples from one of the fixed filters. In another embodiment, for CCALF, the one or more multi-source luma samples include one or more pre-deblocking filtered (pre-DBF) and / or pre-sample adaptive offset (pre-SAO) luma samples in a luma block.

[0014] In one embodiment, one or more luma samples from one of the fixed filters are added to the CCALF filter footprint to obtain a filtered chroma output.

[0015] In one embodiment, the chrominance ALF or CCALF includes high-order input terms. In one embodiment, the high-order input terms correspond to (N 2 -R 2 ), where R is the sample to be processed and N is the target sample. In another embodiment, the high-order input term corresponds to ((sign(NR))*(NR)*(NR)), where R is the sample to be processed and N is the target sample, and sign(NR) returns the sign of (NR). BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A An exemplary adaptive inter / intra video coding system incorporating in-loop processing is shown.

[0017] Figure 1B Shows Figure 1A The corresponding decoder of the encoder in .

[0018] Figure 2 ALF filter shapes for chroma (left) and luma (right) components are shown.

[0019] Figure 3A -D shows the g v (3A), g h (3B), g d1 (3C) and g d2 (3D) Subsampled Laplacian computation.

[0020] Figure 4A The placement of the CC-ALF relative to the other loop filters is shown.

[0021] Figure 4B A diamond filter is shown for chroma sampling.

[0022] Figure 5A -B shows an example of modified block classification at a virtual boundary.

[0023] Fig. 6A -F shows an example of modified ALF filtering for the luma component at various virtual boundary locations.

[0024] Figure 7 3x3 and 5x5 filter shapes are shown applied to the samples before DBF according to JVET-Z0146, with N=24 and N=28 respectively in equation (1).

[0025] Figure 8 A flow chart of an exemplary video coding system utilizing chroma ALF or CCALF with multi-source or high-order taps according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] It is readily understood that, as generally described and illustrated in the accompanying drawings herein, the components of the present invention may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the system and method of the present invention as shown in the figures is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. References to "one embodiment", "an embodiment" or similar language throughout the specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in one embodiment" that appear in various places throughout the specification do not necessarily all refer to the same embodiment.

[0027] In addition, the described features, structures or characteristics can be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the present invention can be practiced without one or more specific details or using other methods, components, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid blurring the various aspects of the present invention. The embodiments shown in the present invention will be better understood with reference to the accompanying drawings, in which the same parts are always represented by the same reference numerals. The following description is intended only by way of example, and simply describes certain selected embodiments of the apparatus and method consistent with the present invention claimed herein.

[0028] Adaptive loop filter in VVC

[0029] In VVC, an adaptive loop filter (ALF) with block-based filter adaptation is applied. For the luminance component, a filter is selected from 25 filters for each 4×4 block based on the direction and activity of the local gradient.

[0030] Filter shape

[0031] Using two diamond filter shapes (such as Figure 2 ). The 7×7 diamond 220 is applied to the luma component and the 5×5 diamond 210 is applied to the chroma components.

[0032] Block Classification

[0033] For the luminance component, each 4×4 block is classified into one of 25 categories. The category index C is based on its directionality D and activity The quantitative value of is derived as follows:

[0034]

[0035] In order to calculate D and First, use the one-dimensional Laplace to calculate the gradient in the horizontal, vertical, and two diagonal directions:

[0036]

[0037] where indices i and j represent the coordinates of the top left sample within the 4×4 block, and R(i,j) represents the reconstructed sample at coordinate (i,j).

[0038] To reduce the complexity of block classification, the subsampled 1D Laplacian calculation is applied in the vertical direction ( Figure 3A ) and horizontal direction ( Figure 3B ).like Figure 3C -D, the same subsampling position is used for gradient calculation in all directions ( Figure 3C g in d1 and Figure 3D g in d2 ).

[0039] The maximum and minimum values ​​of the horizontal and vertical gradients D are set to:

[0040]

[0041] The maximum and minimum values ​​of the two diagonal gradients are set to:

[0042]

[0043] To derive the value of the directionality D, these values ​​are compared with each other and with two thresholds t1 and t2:

[0044] Step 1. If and are all true, then D is set to 0.

[0045] Step 2. If Continue to step 3; otherwise, go to step 4.

[0046] Step 3. If Then D is set to 2; otherwise D is set to 1.

[0047] Step 4. If Then D is set to 4; otherwise D is set to 3.

[0048] The activity value A is calculated as follows:

[0049]

[0050] A is further quantized to a range of 0 to 4 (inclusive), and the quantized value is represented as

[0051] For the chrominance components of the image, no classification is applied.

[0052] Geometric transformation of filter coefficients and clipped values

[0053] Before filtering each 4×4 luma block, geometric transformations such as rotation or diagonal and vertical flipping are applied to the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l) according to the gradient values ​​calculated for that block. This is equivalent to applying these transformations to the samples in the filter support region. The idea is to make different blocks where the ALF is applied more similar by aligning the directionality.

[0054] Introduced three geometric transformations: diagonal, vertical flip and rotation:

[0055] Diagonal:f D (k,l)=f(l,k),c D (k,l)=c(l,k),

[0056] Flip vertically:f V (k,l)=f(k,Kl-1),c V (k,l)=c(k,Kl-1),

[0057] Rotation:f R (k,l)=f(Kl-1,k),c R (k,l)=c(Kl-1,k),

[0058] Where K is the size of the filter and 0≤k,l≤K-1 are the coefficient coordinates, such that position (0,0) is at the upper left corner and position (K-1,K-1) is at the lower right corner. The transform is applied to the filter coefficients f(k,l) and the clipping values ​​c(k,l) according to the gradient values ​​calculated for the block. The relationship between the transform and the four gradients in the four directions is summarized in the following table.

[0059] Table 1. Mapping of gradients and transformations computed for a block

[0060] Gradient Value Transform <![CDATA[g d2 <g d1 And g h <g v ]]> No change <![CDATA[g d2 <g d1 And g v <g h ]]> diagonal <![CDATA[g d1 <g d2 And g h <g v ]]> Flip Vertically <![CDATA[g d1 <g d2 And g v <g h ]]> Rotation

[0061] Filtering

[0062] At the decoder side, when ALF is enabled for CTB, every sample R(i,j) in the CU is filtered to generate the sample value R′(i,j) as follows,

[0063]

[0064] Where f(k,l) represents the decoded filter coefficients, K(x,y) is the clipping function, and c(k,l) represents the decoded clipping parameters. The variables k and l vary between –L / 2 and L / 2, where L represents the filter length. The clipping function K(x,y)=min(y,max(-y,x)) corresponds to the function Clip3(-y,y,x). The clipping operation introduces nonlinearity, making the ALF more efficient by reducing the impact of neighboring sample values ​​that differ too much from the current sample value.

[0065] Cross Component Adaptive Loop Filter (CCALF or CC-ALF)

[0066] CC-ALF uses the luma sample values ​​to refine each chroma component by applying an adaptive linear filter to the luma channel and then using the output of this filtering operation for chroma refinement. Figure 4A A system-level diagram of the CC-ALF process relative to the SAO, luma ALF, and chroma ALF processes is provided. Figure 4A As shown, each color component (i.e., Y, Cb, and Cr) is processed by its respective SAO (i.e., SAO luma 410, SAO Cb 412, and SAO Cr 414). After SAO, ALF luma 420 is applied to the SAO processed luma, while ALF chroma 430 is applied to the SAO processed Cb and Cr. However, there are cross-component terms from luma to chroma components (i.e., CC-ALF Cb 422 and CC-ALF Cr 424). The output from the cross-component ALF is added to the output from ALF chroma 430 (using adders 432 and 434, respectively).

[0067] The filtering in CC-ALF is done by applying linear, diamond filters (e.g. Figure 4B This is accomplished by applying filters 440 and 442 in FIG. 4 to the luminance channel. Figure 4B In the figure, the hollow circles represent luma samples, while the dot-filled circles represent chroma samples. A filter is used for each chroma channel, and its operation is expressed as:

[0068]

[0069] where (x, y) is the chrominance component i whose position is refined, (x Y ,y Y ) is the brightness position based on (x,y), S i is the filter support area in the luminance component, and c i (x0,y0) represents the filter coefficients.

[0070] like Figure 4B As shown, the luma filter support is the area concatenated with the current chroma sample after taking into account the spatial scaling factor between the luma and chroma planes.

[0071] In the VVC reference software, the CC-ALF filter coefficients are calculated by minimizing the mean squared error of each chroma channel relative to the original chroma content. To achieve this goal, the VTM (VVC Test Model) algorithm uses a coefficient derivation process similar to that used for chroma ALF. Specifically, a correlation matrix is ​​derived and the coefficients are calculated using a Cholesky decomposition solver to try to minimize the mean squared error metric. When designing the filters, up to 8 CC-ALF filters can be designed and transmitted per picture. The resulting filters are then indicated for each of the two chroma channels based on the CTU.

[0072] Additional features of CC-ALF include:

[0073] The design adopts a 3x4 diamond shape with 8 taps.

[0074] Seven filter coefficients are transmitted in the APS.

[0075] Each transmitted coefficient has a 6-bit dynamic range and is limited to power-of-2 values.

[0076] • Deriving the eighth filter coefficient at the decoder such that the sum of the filter coefficients is equal to zero.

[0077] APS can be referenced in the slice header.

[0078] CC-ALF controls filter selection at CTU level for each chroma component

[0079] The boundary padding of the horizontal virtual border uses the same memory access pattern as the luma ALF.

[0080] As an additional feature, the reference encoder can be configured to enable some basic subjective tuning via a configuration file. When enabled, VTM will weaken the application of CC-ALF in areas that are close to mid-grey or contain a lot of luminance high frequencies when using a high QP codec. Algorithmically, this is achieved by disabling CC-ALF in a CTU when any of the following conditions are true:

[0081] The slice QP value minus 1 is less than or equal to the base QP value.

[0082] The number of chroma samples with local contrast greater than (1<<(bitDepth–2))–1 exceeds the CTU height, where local contrast is the difference between the maximum and minimum luma sample values ​​within the filter support area.

[0083] More than a quarter of the chroma samples are in the range between (1<<(bitDepth–1))–16 and (1<<(bitDepth–1))+16

[0084] The motivation for this feature is to provide some assurance that CC-ALF does not amplify artifacts introduced earlier in the decoding path (this is mainly due to the fact that VTM does not currently explicitly optimize for chroma subjective quality). It is expected that alternative encoder implementations may not use this feature or include alternative strategies suitable for their encoding characteristics.

[0085] Filter parameter messaging

[0086] ALF filter parameters are signaled in Adaptation Parameter Sets (APSs). In one APS, up to 25 sets of luma filter coefficients and clipping value indices and up to 8 sets of chroma filter coefficients and clipping value indices can be signaled. To reduce bit overhead, filter coefficients of different classifications of luma components can be merged. In the slice header, the index of the APS for the current slice is signaled.

[0087] The clipping value index decoded from the APS allows the clipping values ​​to be determined using the clipping value tables for both luma and chroma components. These clipping values ​​depend on the internal bit depth. More precisely, the clipping values ​​are obtained by the following formula:

[0088] AlfClip = {round(2 B-α*n )forn∈[0..N-1]}

[0089] Where B is equal to the internal bit depth, α is a predefined constant value equal to 2.35, and N is equal to 4, which is the number of clipping values ​​allowed in VVC. AlfClip is then rounded to the nearest value in the power of 2 format.

[0090] In the slice header, up to 7 APS indices can be signaled to specify the luma filter set to be used for the current slice. The filtering process can be further controlled at the CTB level. A flag is always signaled to indicate whether the ALF is applied to the luma CTB. The luma CTB can select a filter set among 16 fixed filter sets and a filter set from the APS. A filter set index is signaled for the luma CTB to indicate which filter set is applied. The 16 fixed filter sets are predefined and hard-coded in the encoder and decoder.

[0091] For chroma components, the APS index is signaled in the slice header to indicate the chroma filter set used for the current slice. At the CTB level, if more than one chroma filter set exists in the APS, the filter index is signaled for each chroma CTB.

[0092] The filter coefficients are quantized with a norm equal to 128. To limit the multiplication complexity, bitstream consistency is applied so that the coefficient values ​​of non-center positions should be in the range of -27 to 27-1 (inclusive). The center position coefficient is not signaled in the bitstream and is considered equal to 128.

[0093] Virtual boundary filtering process for line buffer reduction

[0094] In VVC, in order to reduce the line buffer requirements of ALF, modified block classification and filtering is adopted for samples near the horizontal CTU boundary. To this end, as shown in Figure 5A-B, a virtual boundary is defined as a line by shifting the horizontal CTU boundary by "N" samples, where N is equal to 4 for the luma component and N is equal to 2 for the chroma component.

[0095] The modified block classification is applied to the luminance component, such as Figure 5A -B. For the virtual boundary ( Figure 5A The 4x4 block ( Figure 5A The one-dimensional Laplace gradient computation (shown as the dot-filled area in ) uses only samples above the imaginary boundary (e.g., Figure 5A , fill line K with line J). Similarly, for virtual boundaries (e.g. Figure 5B The 4x4 block below line 540 in Figure 5B ), using only samples below a fictitious boundary (e.g., Figure 5B By taking into account the reduced number of samples used in the one-dimensional Laplace gradient calculation, the quantization of the activity value A is scaled accordingly. The CTU boundary is shown as Figure 5A Lines 510 and 512 and Figure 5B Line 530 in.

[0096] For the filtering process, a symmetric padding operation at the virtual boundary is used for both luma and chroma components. Fig. 6A -F, when the filtered sample is below the virtual boundary, the adjacent samples above the virtual boundary (i.e., outside the virtual boundary area) are filled. At the same time, the corresponding samples on the other side are also filled symmetrically. For example, Fig. 6AIn , the sample in the top line (i.e., sample c0) is above the virtual boundary 610, and the sample is filled with the line below. For the sample in the bottom line (i.e., sample c0), the samples in the bottom line are symmetrically filled with the samples of the upper line. Figure 6B In the example, the virtual border 620 is located just above the bottom line. Since symmetrical padding is used, the padding process is similar to Fig. 6A The same as in. Figure 6C -F marks examples of virtual boundaries (lines 630-660) at different locations. Symmetric padding can be applied similarly.

[0097] Unlike the symmetrical padding used at horizontal CTU boundaries, a simple padding process is applied to slice, tile, and sub-picture boundaries when filters across the boundaries are disabled. A simple padding process is also applied at picture boundaries. The padded samples are used for classification and filtering processes. To compensate for extreme padding when filtering samples above or below the virtual boundary, the filter strength for luma and chroma is reduced in these cases by increasing the right shift by 3.

[0098] Adaptive loop filter in ECM

[0099] ALF Simplification

[0100] Removed ALF gradient subsampling and ALF virtual edge processing. Block size for classification reduced from 4x4 to 2x2. Luma and chroma filter sizes (signaled as ALF coefficients) increased to 9x9.

[0101] ALF with fixed filter

[0102] To filter the luminance samples, three different classifiers (C0, C1, and C2) and three different sets of filters (F0, F1, and F2) are used. Sets F0 and F1 contain fixed filters whose coefficients are trained for classifiers C0 and C1. The filter coefficients in F2 are signaled. Given a sample, the filter is filtered using set F0. i Which filter in is used by the classifier C i The classification C assigned to this sample i Decide.

[0103] Filtering

[0104] First, two 13x13 diamond fixed filters F0 and F1 are applied to derive two intermediate samples R0(x,y) and R1(x,y). Afterwards, F2 is applied to R0(x,y), R1(x,y) and the adjacent samples to derive the filtered samples:

[0105]

[0106] where f i,j is the difference in amplitude between the adjacent sample and the current sample R(x,y), g i YesR i-20 The difference between (x,y) and the current sample. The signal filter coefficient c i ,i=0,…21.

[0107] Classification

[0108] Based on the directionality D i and activities Classify C i Assign to each 2x2 block:

[0109]

[0110] Among them, M D,i Indicates directionality D i The total number of .

[0111] As in VVC, a one-dimensional Laplacian operator is used to calculate the horizontal, vertical, and two diagonal gradient values ​​for each sample. The sum of the sampled gradients within a 4×4 window covering the target 2×2 block is used for classifier C0, while the sum of the sampled gradients within a 12×12 window is used for classifiers C1 and C2. The sum of the horizontal, vertical, and two diagonal gradients is expressed as and Directionality D i By comparison

[0112]

[0113] is determined with a set of thresholds. Directivity D2 is derived as in VVC using thresholds 2 and 4.5. For D0 and D1, first calculate the horizontal / vertical edge strength and diagonal edge strength . Use threshold Th = [1.25, 1.5, 2, 3, 4.5, 8]. If The edge strength is 0; otherwise, is satisfied The largest integer. The edge strength is 0; otherwise, is satisfied The maximum integer of . That is, when horizontal / vertical edges dominate, use Table 2A to derive D i Otherwise, the diagonal edges dominate, and D is derived by using Table 2B i .

[0114] Table 2A. and Mapping to D i

[0115]

[0116] Table 2B. and Mapping to D i

[0117]

[0118]

[0119] To obtain The sum of the vertical and horizontal gradients A i Mapped to the range from 0 to n, where for , n is equal to 4, for and n is equal to 15.

[0120] In ALF_APS, up to 4 luminance filter sets are signaled, each set may have up to 25 filters.

[0121] Adaptive loop filter with multiple sources

[0122] In JVET-Z0146 (Nan Hu et al., "AHG12: Using samples before deblocking filterfor adaptive loop filter"), Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC, 26th Meeting, April 20-29, 2022, by teleconference, document: JVET-Z0146), for the luminance component, it is recommended that the samples before the deblocking filter (DBF) be used for ALF as well. Specifically, the derivation of the filtered samples is as follows:

[0123]

[0124] where h i,j is the difference in amplitude between the adjacent sample before DBF and the current sample R(x,y). Two filter shapes are proposed, namely 3x3(710) and 5x5(720) diamonds, as shown in Figure 7 shown.

[0125] In the Adaptation Parameter Set (APS), a flag is signaled to indicate whether the samples before DBF are used for ALF. In JVET-Z0146, this flag is always set to true at the encoder.

[0126] In the present invention, techniques for improving ALF performance and / or simplifying processing / storage are disclosed as follows. Specifically, a technique for using virtual boundary processing for ALF processing is disclosed to reduce line buffer usage in ALF. Various methods are proposed below.

[0127] In the present invention, when neighboring samples are unavailable, the padding process used in the filtering process is replaced by setting the difference between the neighboring samples and the samples to be processed to zero. In one example, the difference between the neighboring samples and the samples to be processed is set to zero only for those neighboring samples that are unavailable. That is, this is an asymmetric process. In another example, the process of setting the difference between the neighboring samples and the samples to be processed to zero is applied to two corresponding symmetric positions. That is, the filter footprint is modified, such as Fig. 6A -F, but changes the padding above and below to set the difference between the adjacent sample and the sample to be processed to zero. This method can be used for luma ALF, chroma ALF, and / or CCALF.

[0128] In another method of luma ALF, in addition to the samples before ALF, it also includes multiple sources in the filter footprint. The multiple sources can be samples before the deblocking filter, samples before SAO, samples after applying the ALF fixed filter, reconstructed residuals after inverse transform, and / or samples before the reconstruction stage (e.g., inter-frame / intra-frame predictors). In this case, in order to further reduce the use of the buffer, the ALF virtual boundary process is also applied to these multiple sources. For example, the samples before the deblocking filter are used in the ALF. If the samples required before the deblocking filter are not available (e.g., the samples before the deblocking filter are on the other side of the virtual boundary), a filling process is used to avoid accessing these samples. The filling process can be asymmetric or symmetric, such as used in VVC. In another embodiment, the filling process is replaced by setting the difference between the samples required before the deblocking filter and the samples to be processed to zero. In another embodiment, if one of the samples required before the deblocking filter is not available, the filter taps of the samples before the deblocking filter are removed. In another embodiment, if one of the required samples before the deblocking filter is not available, the filter taps for the sample before the deblocking filter are reduced to a single tap corresponding to the position of the sample to be processed. In one embodiment, the virtual boundary processing for the sample before the ALF is the same as the virtual boundary processing for multiple sources. That is, the same virtual boundary processing is applied to all input sources of the luma ALF.

[0129] In one approach, multiple sources are also utilized in the chroma ALF and / or CCALF to further improve codec performance. In one example, chroma samples before the deblocking filter and chroma samples before the SAO are added to the filter footprint of the chroma ALF. In another example, luma samples before the deblocking filter and luma samples before the SAO are added to the filter footprint of the CCALF.

[0130] In one embodiment, multiple sources can also come from different components. For example, for luma ALF, chroma samples before deblocking filter can be added to luma ALF filter footprint. In another example, luma samples before deblocking filter, chroma samples before deblocking filter, luma samples before SAO, and chroma samples before SAO are added to luma ALF filter footprint. In another example, luma samples before deblocking filter and luma samples before SAO are added to the filter footprint of chroma ALF.

[0131] In another embodiment, the filter footprint of the chroma ALF may include two chroma components. That is, when the ALF is applied to the Cb component, the Cr samples before the ALF are also used for the chroma ALF. The Cr samples before the ALF refer to any sample type before the ALF, such as the Cr samples immediately before the ALF (i.e., pre-ALF), immediately before the SAO (i.e., pre-SAO), or immediately before the deblocking filter (i.e., pre-DBF).

[0132] In one embodiment, multiple sources can come from the intermediate ALF filtering results. For example, luma samples after applying different fixed filters can be added to the luma ALF filter footprint. In another example, luma samples after applying different fixed filters can be added to the CCALF filter footprint.

[0133] In the above-mentioned proposed method, one or more filter taps of multiple sources can be one or more high-degree parameters. For example, considering the sample to be processed R and the target sample N, instead of using (NR), the square difference (N 2 -R 2 ) as an additional tap. In another example, the input may be sign(NR)*((NR)*(NR)), where sign(x) is used to return "+1" when x is non-negative and "-1" when x is negative.

[0134] In the above proposed method, when multiple sources are introduced in the ALF, nonlinear operation (ie, limiting operation) can also be applied together.

[0135] Any of the above multi-source methods for loop filters (e.g., chroma ALF / CCALF methods) can be implemented in an encoder and / or decoder. For example, any of the proposed multi-source ALF methods can be implemented in a loop filter module (e.g., Figure 1A and Figure 1B Alternatively, any of the proposed methods may be implemented as an inter-frame coding module coupled to an encoder and / or a motion compensation module, a merge candidate derivation module of a decoder. The ALF method may also be implemented using executable software or firmware code stored on a medium (e.g., a hard disk or flash memory) for a CPU (central processing unit) or a programmable device (e.g., a DSP (digital signal processor) or an FPGA (field programmable gate array)).

[0136] Figure 8 A flowchart of an exemplary video encoding and decoding system using a chroma ALF or CCALF with multi-source or high-order taps according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as executable program codes on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, a reconstructed pixel is received in step 810, wherein the reconstructed pixel includes a current block, and the current block includes a luminance block and one or more chroma blocks. In step 820, a filtered chroma output is derived from a chroma ALF or CCALF (cross-component ALF) of a current chroma sample in one of the one or more chroma blocks, wherein the chroma ALF includes one or more multi-source chroma samples or luminance samples from the current block in the first coverage area of ​​the chroma ALF, or wherein the CCALF includes one or more multi-source luminance samples from the luminance block in the second coverage area of ​​the CCALF. A first chroma block of filtered reconstruction is provided in step 830, wherein the first chroma block of filtered reconstruction includes a filtered chroma output.

[0137] The flowchart shown is intended to illustrate an example of video coding according to the present invention. Those skilled in the art may modify each step, rearrange the steps, split the steps, or combine the steps to implement the present invention without departing from the spirit of the present invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples of implementing embodiments of the present invention. A skilled person may practice the present invention by replacing syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention.

[0138] The above description is given to enable those skilled in the art to practice the present invention provided in the context of specific applications and their requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but should be given the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are shown in order to provide a thorough understanding of the present invention. However, it will be appreciated by those skilled in the art that the present invention can be put into practice.

[0139] Embodiments of the present invention as described above can be implemented with various hardware, software codes or a combination of the two. For example, embodiments of the present invention can be one or more circuits integrated into a video compression chip or a program code integrated into a video compression software to perform the processing described herein. Embodiments of the present invention can also be program codes executed on a digital signal processor (DSP) to perform the processing described herein. The present invention can also relate to multiple functions performed by a calculator processor, a digital signal processor, a microprocessor or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software codes or firmware codes that define the specific methods embodied by the present invention. Software codes or firmware codes can be developed in different programming languages ​​and different formats or styles. Software codes can also be compiled for different target platforms. However, different code formats, styles and languages ​​of software codes and other ways of configuring codes to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0140] The present invention may be implemented in other specific forms without departing from its spirit or essential features. The described examples should be considered in all respects as merely illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than the foregoing description. All changes within the meaning and scope of the equivalents of the claims are included within their scope.

Claims

1. A method for ALF processing for reconstructing video, the method comprising: Receiving reconstructed pixels, wherein the reconstructed pixels include a current block, and the current block includes a luma block and one or more chroma blocks; deriving a filtered chroma output from a chroma ALF or a cross-component ALF for a current chroma sample in one of the one or more chroma blocks, wherein the chroma ALF includes one or more multi-source chroma samples or luma samples for the current block in a first footprint of the chroma ALF, or wherein the cross-component ALF includes one or more multi-source luma samples for the luma block in a second footprint of the cross-component ALF; as well as A filtered reconstructed first chroma block is provided, wherein the filtered reconstructed first chroma block includes the filtered chroma output.

2. The method according to claim 1, characterized in that For the chroma ALF, the one or more multi-source chroma samples include one or more pre-DBF and / or pre-sample adaptive offset chroma samples in the one or more chroma blocks.

3. The method according to claim 1, characterized in that For the chroma ALF, the one or more multi-source chroma samples include one or more chroma samples preceding any ALF type in another of the one or more chroma blocks.

4. The method according to claim 1, characterized in that For the chroma ALF, the one or more multi-source luma samples include one or more pre-DBF and / or pre-sample adaptive offset luma samples in the luma block.

5. The method according to claim 1, characterized in that For the cross-component ALF, the one or more multi-source luma samples include one or more luma output samples from one of the fixed filters.

6. The method according to claim 1, characterized in that For the cross-component ALF, the one or more multi-source luma samples include one or more pre-DBF and / or pre-sampled adaptive offset luma samples in the luma block.

7. The method according to claim 1, characterized in that One or more luma samples from one of the fixed filters are added to the cross-component ALF filter footprint for the filtered chroma output.

8. The method according to claim 1, characterized in that The chroma ALF or the cross-component ALF includes high-order input terms.

9. The method according to claim 8, characterized in that This high-order input term corresponds to (N 2 -R 2 ), where R is the sample to be processed and N is the target sample.

10. The method according to claim 8, characterized in that This high-order input term corresponds to ((sig n(NR))*(NR)*(NR)), where R is the sample to be processed, N is the target sample, and sign(NR) returns the sign of (NR).

11. An apparatus for ALF processing for reconstructing video, the apparatus comprising one or more electronic circuits or processors arranged to: Receiving reconstructed pixels, wherein the reconstructed pixels include a current block, and the current block includes a luma block and one or more chroma blocks; deriving a filtered chroma output from a chroma ALF or a cross-component ALF for a current chroma sample in one of the one or more chroma blocks, wherein the chroma ALF includes one or more multi-source chroma samples or luma samples for the current block in a first footprint of the chroma ALF, or wherein the cross-component ALF includes one or more multi-source luma samples for the luma block in a second footprint of the cross-component ALF; and A filtered reconstructed first chroma block is provided, wherein the filtered reconstructed first chroma block includes the filtered chroma output.

Images