Method of processing color image and apparatus for processing coded video

Abstract

A method of processing a color image and an apparatus for processing a coded video. According to the method, input data associated with a current block is received, including a first color block and a second color block, where the first color block includes first color samples and the second color block includes second color samples. One or more target second color samples are derived from a CCM applied to one or more cross component model (CCM) input first color samples, or one or more target second color samples are derived by applying one or more interpolation filters to one or more interpolation input first color samples or second color samples, the one or more target second color samples are derived at one or more non-integer positions. One or more filtered second color samples are generated by applying a target adaptive loop filter (ALF), wherein the ALF uses filtered input samples comprising one or more filtered input second color samples and the one or more target second color samples.

Description

[0001]

Cross-reference

[0002] This application is a non-provisional application based on and claims priority to U.S. Provisional Patent Application No. 63 / 478,705, filed January 6, 2023, and U.S. Provisional Patent Application No. 63 / 439,226, filed January 16, 2023. The aforementioned U.S. Provisional Patent Applications are incorporated herein by reference in their entirety.

Technical field

[0003] The present invention relates to a video coding and decoding system, and in particular to a new technique for generating model-based tap inputs for an adaptive loop filter (ALF). [Background Technology]

[0004] Versatile Video Codec (VVC) is the latest international video codec standard developed by the ITU-T Video Codec Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG) Joint Video Experts Team (JVET). The standard was published as ISO standard ISO / IEC 23090-3:2021, Information technology — Coded representation of immersive media — Part 3: Versatile video codec, in February 2021. VVC builds on its predecessor, High Efficiency Video Codec (HEVC), by adding additional codec tools to improve codec efficiency and handle various video sources, including three-dimensional (3D) video signals.

[0005] Figure 1AAn exemplary adaptive inter / intra video coding system incorporating loop processing is shown. For intra prediction 110, prediction data is derived based on previously encoded video data from the current picture. For inter prediction 112, motion estimation (ME) is performed at the encoder, and motion compensation (MC) is performed based on the ME results to provide prediction data derived from other pictures and motion data. A switch 114 selects either intra prediction 110 or inter prediction 112, and the selected prediction data is provided to an adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by a transform (T) 118 and then quantized (Q) 120. The transformed and quantized residual is then encoded by an entropy encoder 122 for inclusion in a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with additional information, such as the motion and codec mode associated with intra and inter prediction, as well as parameters associated with the loop filter applied to the underlying image region. Additional information related to intra prediction 110, inter prediction 112 and loop filter 130 is as follows: Figure 1A The image is provided to the entropy encoder 122. When using inter-frame prediction mode, the reference picture or pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed through inverse quantization (IQ) 124 and inverse transformation (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in the reference picture buffer (RPB) 134 and used for prediction of other frames.

[0006] like Figure 1A As shown, the input video data undergoes a series of processing in the encoding system. The reconstructed video data from the reconstruction 128 may be subjected to various impairments due to the series of processing. Therefore, a loop filter 130 is typically applied to the reconstructed video data before the reconstructed video data is stored in the reference picture buffer 134 to improve the video quality. For example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF) may be used. The loop filter information may need to be merged into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be merged into the bitstream. In Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in the example is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to a High Efficiency Video Codec (HEVC) system, VP8, VP9, H.264, or VVC.

[0007] like Figure 1B As shown, the decoder can use the same or partially the same functional modules as the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses an entropy decoder (EntropyDecoder) 140 to decode the video bitstream into quantized transform coefficients and required codec information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 at the decoder side does not need to perform a pattern search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from the entropy decoder 140, without the need for motion estimation.

[0008] According to VVC, the input picture is divided into non-overlapping square areas called codec tree units (CTUs), similar to HEVC. Each CTU can be divided into one or more smaller-sized codec units (CUs). The generated CU partitions can be square or rectangular in shape. In addition, VVC divides the CTU into prediction units (PUs), which serve as units for applying prediction processes such as inter-frame prediction, intra-frame prediction, etc.

[0009] Adaptive loop filter in VVC

[0010] In VVC, an adaptive loop filter (ALF) with block-based filter adaptability is applied. For the luminance component, a filter is selected from 25 filters for each 4×4 block based on the direction and activity of the local gradient.

[0011] Filter shape

[0012] Two diamond filter shapes were used (e.g. Figure 2 ). The 7×7 diamond shape 220 is applied to the luma component, and the 5×5 diamond shape 210 is applied to the chroma components.

[0013] Block Classification

[0014] For the luminance component, each 4×4 block is classified into one of 25 categories. The category index C is based on the quantized values of its directionality D and activity The derivation is as follows:

[0015]

[0016] To calculate D and First, use the one-dimensional Laplace to calculate the gradient in the horizontal, vertical, and two diagonal directions:

[0017]

[0018] Here, the indices i and j refer to the coordinates of the upper left corner sample in the 4×4 block, and R(i,j) represents the reconstructed sample at coordinate (i,j).

[0019] To reduce the complexity of block classification, subsampled 1D Laplacian calculations are applied in the vertical direction ( Figure 3A ) and horizontal direction ( Figure 3B ).like Figure 3C -D, the same subsampling position is used for gradient calculation in all directions ( Figure 3C g in d1 and Figure 3D t in d2 ).

[0020] Then, the maximum and minimum values of the horizontal and vertical gradients are set to:

[0021]

[0022] The maximum and minimum values of the two diagonal gradients are set to:

[0023]

[0024] To derive the value of the directionality D, these values are compared with each other and with two thresholds t1 and t2:

[0025] Step 1. If and If both are true, D is set to 0.

[0026] Step 2. If Continue with step 3; otherwise, continue with step 4.

[0027] Step 3. If Then D is set to 2; otherwise D is set to 1.

[0028] Step 4. If Then D is set to 4; otherwise D is set to 3.

[0029] The activity value A is calculated as:

[0030]

[0031] A is further quantized to the range of 0 to 4 (inclusive), and the quantized value is expressed as

[0032] No classification is performed on the chrominance components in the image.

[0033] Geometric transformation of filter coefficients and clipping values

[0034] Before filtering each 4×4 luma block, the filter coefficients f(k,l) and the corresponding filter clipping values c(k,l) are geometrically transformed, such as rotated or diagonally and vertically flipped, according to the gradient values calculated for that block. This is equivalent to applying these transformations to the samples within the filter support region. The purpose is to make different blocks to which the ALF is applied more similar by aligning their directionality.

[0035] Three geometric transformations are introduced, including diagonal, vertical flip and rotation:

[0036] Diagonal: f D (k,l)=f(l,k),c D (k,l)=c(l,k),

[0037] Flip vertically: f V (k,l)=f(k,Kl-1),c V (k,l)=c(k,Kl-1),

[0038] Rotation: f R (k,l)=f(Kl-1,k),c R (k, l) = c(Kl-1, k), where K is the filter size and 0 ≤ k, l ≤ K-1 are the coefficient coordinates, such that position (0, 0) is at the top left corner and position (K-1, K-1) is at the bottom right corner. The filter coefficients f(k, l) and the cropping values c(k, l) are transformed based on the gradient values calculated for that block. The relationship between the transformation and the four gradients in the four directions is summarized in the table below.

[0039] Table 1. Mapping of gradients and transformations for block computation

[0040]

[0041] Filtering process

[0042] At the decoder side, when ALF (Adaptive Loop Filter) is enabled for CTB (Coding Tree Block), each sample R(i,j) within the CU (Coding Unit) is filtered, and the result is the sample value R′(i,j), as shown below,

[0043]

[0044] Where f(k,l) represents the decoded filter coefficients, K(x,y) is the clipping function, and c(k,l) represents the decoded clipping parameters. The variables k and l vary between –L / 2 and L / 2, where L represents the filter length. The clipping function K(x,y) = min(y,max(-y,x)) corresponds to the function Clip3(-y,y,x). The clipping operation introduces nonlinearity, making the ALF more efficient by reducing the influence of neighboring samples that differ significantly from the current sample value.

[0045] Cross-Component Adaptive Loop Filter (CC-ALF)

[0046] CC-ALF (or CCALF) uses the luma sample values to refine each chroma component by applying an adaptive linear filter to the luma channel, and then uses the output of this filtering operation for chroma refinement. Figure 4A A system-level diagram of the CC-ALF process relative to the SAO (Sample Adaptive Offset), Luma ALF, and Chroma ALF processes is provided. Figure 4A As shown, each color component (i.e., Y, Cb, and Cr) is processed by its corresponding SAO (i.e., SAO Luma 410, SAO Cb 412, and SAO Cr 414). After SAO, ALF Luma 420 is applied to the SAO-processed luma, and ALF Chroma 430 is applied to the SAO-processed Cb and Cr. However, there is a cross-component term from luma to chroma components (i.e., CC-ALF Cb 422 and CC-ALF Cr 424). The output of the cross-component ALF is summed with the output of ALF Chroma 430 (using adders 432 and 434, respectively).

[0047] In CC-ALF, a linear, diamond filter (e.g. Figure 4B Filters 440 and 442 in the filter are used to complete the filtering. Figure 4B In the figure, the hollow circles represent luminance samples and the solid circles represent chrominance samples. A filter is used for each chrominance channel, and the operation is expressed as:

[0048]

[0049] where (x,y) is the position of the chrominance component i being refined, (x Y ,yY ) is the brightness position based on (x,y), S i is the filter support area in the luminance component, c i (x0,y0) represents the filter coefficients.

[0050] like Figure 4B As shown, the luma filter support region is the region that coincides with the current chroma sample after taking into account the spatial scaling factor between luma and planes.

[0051] In the VVC (Versatile Video Coding) reference software, CC-ALF filter coefficients are calculated by minimizing the mean squared error (MSE) of each chroma channel relative to the original chroma content. To this end, the VTM (VVC Test Model) algorithm uses a coefficient derivation process similar to that used for the chroma ALF. Specifically, a correlation matrix is derived, and the coefficients are calculated using a Cholesky decomposition solver to minimize the MSE metric. When designing the filters, up to eight CC-ALF filters can be designed and transmitted per image. The resulting filters are then indicated for each of the two chroma channels on a CTU (Coding Tree Unit) basis.

[0052] Additional features of CC-ALF include:

[0053] The design uses a 3x4 diamond shape with 8 taps.

[0054] The seven filter coefficients are transmitted in an APS (Adaptive Parameter Set).

[0055] Each transmitted coefficient has a 6-bit dynamic range and is limited to power-of-two values.

[0056] The eighth filter coefficient is derived at the decoder such that the sum of the filter coefficients is equal to zero.

[0057] APS can be referenced in the title.

[0058] CC-ALF filter selection is controlled per chroma component at the CTU level.

[0059] Border filling for horizontal virtual borders uses the same memory access pattern as luma ALF.

[0060] As an additional feature, the reference encoder can enable some basic subjective tuning via the configuration file. When enabled, VTM will reduce the application of CC-ALF in areas encoded with high QP and close to mid-gray or containing a lot of luminance high frequencies. Algorithmically, this is achieved by disabling the application of CC-ALF in a CTU when any of the following conditions are true:

[0061] The slice QP value minus 1 is less than or equal to the base QP value.

[0062] The number of chroma samples with local contrast greater than (1<<(bitDepth–2))–1 exceeds the CTU height, where local contrast is the difference between the maximum and minimum luma sample values within the filter support area.

[0063] More than a quarter of the chroma samples are between (1<<(bitDepth–1))–16 and (1<<(bitDepth–1))+16.

[0064] The motivation for this feature is to provide some assurance that CC-ALF does not amplify artifacts introduced earlier in the decoding path (primarily because VTM currently does not explicitly optimize for subjective quality of chroma). It is expected that alternative encoder implementations may not use this feature, or adopt alternative strategies appropriate to their encoding characteristics.

[0065] Filter parameter signaling

[0066] ALF filter parameters are signaled in an Adaptive Parameter Set (APS). In one APS, up to 25 sets of luma filter coefficients and cropping value indices, and up to eight sets of chroma filter coefficients and cropping value indices can be signaled. To reduce bit overhead, filter coefficients of different classifications of the luma component can be merged. The index of the APS used for the current slice is signaled in the slice header.

[0067] The clipping value index decoded from the APS allows the use of the clipping value tables for the luma and chroma components to determine the clipping value. These clipping values depend on the internal bit depth. More precisely, the clipping value is obtained by the following formula:

[0068] AlfClip={round(2 B-α*n )forn∈[0..N-1]}

[0069] Where B is equal to the internal bit depth, ( is a predefined constant value equal to 2.35, and N is equal to 4, which is the number of crop values allowed in VVC. AlfClip is then rounded to the nearest power of 2 value.

[0070] In the slice header, up to seven APS indices can be signaled to specify the luma filter set used for the current slice. The filtering process can be further controlled at the CTB level. A flag is always signaled to indicate whether the ALF is applied to the luma CTB. The luma CTB can select a filter set from 16 fixed filter sets and a filter set from the APS. A filter set index is signaled for the luma CTB to indicate which filter set to apply. These 16 fixed filter sets are predefined and hard-coded in both the encoder and decoder.

[0071] For chroma components, an APS index is signaled in the slice header to indicate the chroma filter set used for the current slice. At the CTB level, if there are multiple chroma filter sets in the APS, a filter index is signaled for each chroma CTB.

[0072] The filter coefficients are quantized to a standard value of 128. To limit the multiplication complexity, bitstream consistency is applied so that the values of coefficients in non-center positions shall be between -27 and 27-1, inclusive. The center position coefficients are not signaled in the bitstream and are treated as equal to 128.

[0073] Adaptive loop filter in ECM

[0074] In recent video standards development, an ALF that is more advanced than the ALF in VVC has been disclosed. The state of the codec algorithm developed is described in the Enhanced Compression Model (ECM) and is updated during each meeting (e.g., ECM-6, Muhammed Coban et al., "Algorithmic Description of Enhanced Compression Model 6 (ECM 6)", 27th Meeting, by teleconference, July 13-22, 2022, document: JVET-AA2025). The ALF filtering process according to ECM is described below.

[0075] ALF simplification

[0076] ALF gradient subsampling and ALF virtual boundary processing are removed. The block size for classification is reduced from 4x4 to 2x2. The filter size for luma and chroma of ALF coefficient signaling is increased to 9x9.

[0077] ALF and fixed filter

[0078] To filter a luminance sample, three different classifiers (C0, C1 and C2) and three different sets of filters (F0, F1 and F2) are used. The F0 and F1 sets contain fixed filters whose coefficients are trained for the classifiers C0 and C1. The filter coefficients in F2 are signaled. Which filter set Fi is used for a given sample is determined by the class C assigned to that sample using the classifier Ci. i Decide.

[0079] Filtering

[0080] First, two 13x13 diamond fixed filters F0 and F1 are applied to derive two intermediate samples R0(x,y) and R1(x,y). Afterwards, F2 is applied to R0(x,y), R1(x,y) and the neighboring samples to derive the filtered samples as

[0081]

[0082] where f i,jis the cropped difference between the neighboring sample and the current sample R(x,y), g i It is R i-20 The cropped difference between (x,y) and the current sample. Filter coefficient c i ,i=0,…21,is signaled.

[0083] Classification

[0084] Based on the directionality D i and activity Assign a class C to each 2x2 block i :

[0085]

[0086] Among them, M D,i Indicates the total number of directions D i .

[0087] As in VVC, the one-dimensional Laplacian operator is used to calculate the horizontal, vertical and two diagonal gradient values of each sample. The sum of the sample gradients within the 4×4 window covering the target 2×2 block is used for classifier C0, while the sum of the sample gradients within the 12×12 window is used for classifiers C1 and C2. The sum of the horizontal, vertical and two diagonal gradients is expressed as and By Compare with a set of thresholds to determine the directionality D i Directivity D2 is derived as in VVC using thresholds 2 and 4.5. For D0 and D1, first calculate the horizontal / vertical edge strength and diagonal edge strength Use threshold Th=[1.25,1.5,2,3,4.5,8]. If The edge strength is 0; otherwise, is to make If The edge strength is 0; otherwise, is to make The maximum integer of . When horizontal / vertical edges dominate, use Table 2A to derive D i Otherwise, the diagonal edges dominate, and D is derived using Table 2B. i .

[0088] Table 2A. and to D i Mapping

[0089]

[0090] Table 2B. and to D i Mapping

[0091]

[0092]

[0093] In order to obtain The sum of the vertical and horizontal gradients A i Mapped to the range 0 to n, where n is Equal to 4, for and Equal to 15.

[0094] In ALF_APS, up to 4 luminance filter sets can be signaled, and each set can have up to 25 filters.

[0095] Convolutional Cross-Component Model (CCCM)

[0096] In this method, a convolutional cross-component model (CCCM) is applied to predict chroma samples from reconstructed luma samples, similar to the current cross-component linear model (CCLM) approach. When chroma subsampling is used, the reconstructed luma samples are downsampled to match the lower resolution chroma grid. Similar to CCLM, reference samples at the top, left, or top and left are used as templates for model derivation.

[0097] Similarly, similar to CCLM, you can choose to use a single model or a multi-model variant of CCCM. The multi-model variant uses two models, one for samples with above-average luminance reference values and another for the remaining samples (following the principles of CCLM design). For prediction units (PUs) with at least 128 available reference samples, the multi-model CCCM mode can be selected.

[0098] Convolution filter

[0099] The convolutional 7-tap filter consists of a 5-tap plus-shaped spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter consists of a center (C) luma sample co-located with the chroma sample to be predicted and its neighbors above / north (N), below / south (S), left / west (W), and right / east (E), as shown in Figure 5 shown.

[0100] The nonlinear term (denoted as P) is expressed as the square of the center luminance sample C, scaled to the sample value range of the content:

[0101] P=(C*C+midVal)>>bitDepth.

[0102] For example, for 10-bit content, the nonlinear term is calculated as:

[0103] P=(C*C+512)>>10

[0104] The bias term (denoted as B) represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0105] The output of the filter is calculated by convolving the filter coefficients ci with the input value and clipped to the range of valid chroma samples:

[0106] predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B.

[0107] Calculation of filter coefficients

[0108] Filter coefficient c i It is calculated by minimizing the mean square error (MSE) between the predicted and reconstructed chroma samples in the reference region. Figure 6 An example of a reference region is shown, consisting of 6 rows of chroma samples above and to the left of the PU. The reference region is extended to the right by one PU width and downward by one PU height. The region is adjusted to contain only the available samples. The extension of the region (labeled "Extended Region") is to support Figure 5 "Side samples" of the plus-shaped spatial filter, used to fill in unavailable areas.

[0109] MSE minimization is performed by computing the autocorrelation matrix of the luma input and the cross-correlation vector between the luma input and the chroma output. The autocorrelation matrix undergoes an LDL decomposition, and the final filter coefficients are calculated by back substitution. This process roughly follows the calculation of the ALF filter coefficients in ECM, but LDL decomposition is chosen instead of Cholesky decomposition to avoid the use of square root operations.

[0110] The autocorrelation matrix is calculated using the reconstructed values of the luma and chroma samples. These samples are full range (e.g., between 0 and 1023 for 10-bit content), resulting in relatively large values in the autocorrelation matrix. This requires high bit depth operations during model parameter calculations. It is recommended to remove the fixed offsets from the luma and chroma samples of each PU for each model. This reduces the size of the values used in model creation and allows reducing the precision required for fixed-point arithmetic. Therefore, it is recommended to use 16 bits of fractional precision instead of the 22 bits of precision of the original CCCM implementation.

[0111] Reference sample values outside the top left corner of the PU are used as offsets (offsetLuma, offsetCb, and offsetCr) to simplify processing. The sample values used in model creation and final prediction (i.e., luma and chroma in the reference region, and luma in the current PU) are reduced by the following fixed values:

[0112] C'=C–offsetLuma,

[0113] N'=N–offsetLuma,

[0114] S'=S–offsetLuma,

[0115] E'=E–offsetLuma,

[0116] W'=W–offsetLuma,

[0117] P'=nonLinear(C'),

[0118] B=midValue=1<<(bitDepth-1),

[0119] Chroma values are predicted using the following equations, where offsetChroma is equal to offsetCr and offsetCb for the Cr and Cb components, respectively:

[0120] predChromaVal=c0C'+c1N'+c2S'+c3E'+c4W'+c5P'+c6B+offsetChroma.

[0121] To avoid any additional sample-level operations, the luma offset is removed during the luma reference sample interpolation. This can be achieved by replacing the rounding term used in the luma reference sample interpolation with an updated offset containing the rounding term and offsetLuma. The chroma offset can be removed by directly subtracting the chroma offset from the reference chroma samples. As an alternative, the effect of the chroma offset can be removed from the cross-component vector, resulting in the same result. To add the chroma offset back to the output of the convolution prediction operation, the chroma offset is added to the bias term of the convolution model.

[0122] The CCCM model parameter calculation process requires division operations. Division operations are not always considered easy to implement. Division operations are replaced by multiplication (with a scaling factor) and shift operations, where the scaling factor and the number of shifts are calculated based on the denominator, similar to the method used in CCLM parameter calculation.

[0123] Gradient Linear Model (GLM)

[0124] For YUV 4:2:0 color format, the gradient linear model (GLM) method can be used to predict chrominance samples from the gradient of luma samples. Two modes are supported: two-parameter GLM mode and three-parameter GLM mode.

[0125] Compared to CCLM, GLM uses the gradient of luma samples to infer the linear model instead of the downsampled luma values. Specifically, when applying GLM, the input of the CCLM process, i.e., the downsampled luma samples L, is replaced by the luma sample gradient G. The other parts of CCLM (e.g., parameter inference, linear transformation of prediction samples) remain unchanged.

[0126] C=α·G+β

[0127] In the three-parameter GLM, chrominance samples can be predicted using different parameters based on the gradient of luma samples and the downsampled luma values. The model parameters of the three-parameter GLM are derived from the six row and column neighboring samples by minimizing the MSE of the LDL decomposition, similar to the method used in CCCM.

[0128] C=α0·G+α1·L+α2·β

[0129] For signaling, when the cross-component linear model (CCLM) mode is enabled for the current coding unit (CU), two flags are signaled for the Cb and Cr components separately to indicate whether the gradient linear model (GLM) is enabled for each component, or a GLM flag with a shared GLM index is signaled for the Cb and Cr components. If GLM is enabled for a component, a syntax element is further signaled to select multiple gradient filters for gradient calculation ( Figure 7 710-740 in . GLM can be used in combination with the existing CCLM by signaling an additional flag in the bitstream. When this combination is applied, the filter coefficients used to derive the linear model input luminance samples are calculated as a combination of the selected gradient filter of the GLM and the downsampling filter of the CCLM.

[0130] Bitstream signaling

[0131] The use of this mode is signaled via a prediction unit (PU) level flag for CABAC coding. To support this, a new CABAC context is included. When signaling, the convolutional cross-component model (CCCM) is treated as a sub-mode of CCLM. That is, the CCCM flag is signaled only when the intra prediction mode is LM_CHROMA.

[0132] In the present invention, techniques for generating adaptive loop filter (ALF) taps based on a cross-component model are disclosed to improve performance. [Summary of the invention]

[0133] A method and apparatus for generating a cross-component model-based adaptive loop filter (ALF) tap is disclosed. According to the method, input data associated with a current block is received, including a first color block and a second color block, wherein the first color block includes a first color sample and the second color block includes a second color sample. One or more target second color samples are derived based on a cross-component model (CCM) applied to one or more CCM input first color samples, or the one or more target second color samples are derived at one or more non-integer positions by applying one or more interpolation filters to one or more interpolated input first color samples or one or more interpolated input second color samples. One or more filtered second color samples are generated by applying a target adaptive loop filter (ALF), wherein the ALF uses filtered input samples including one or more filtered input second color samples and the one or more target second color samples.

[0134] In one embodiment, the cross-component model corresponds to an ALF type filtering process. In one embodiment, coefficients associated with the cross-component model are derived using reconstructed first color samples and reconstructed second color samples of the adjacent reference region and / or the current block. In one embodiment, the reconstructed first color samples and reconstructed second color samples of the adjacent reference region correspond to reconstructed samples before or after the ALF filtering process. In one embodiment, the adjacent reference region is classified into a plurality of regions, and a cross-component model is derived for each of the plurality of regions. In one embodiment, the one or more target second color samples are derived by applying the one cross-component model according to a category associated with one of the plurality of regions.

[0135] In one embodiment, the one or more CCM input first color samples correspond to reconstructed first color samples before or after an ALF filtering process.

[0136] In one embodiment, the one or more target second color samples are used independently for a target ALF.In one embodiment, the target ALF is separate from a luma ALF, a chroma ALF, or a cross-component ALF (CCALF) filtering process.

[0137] In one embodiment, the cross-component model corresponds to a convolutional cross-component model (CCCM) type of filtering process.

[0138] In one embodiment, the one or more target second color samples are used as reconstructed samples after applying the ALF filtering process.

[0139] In one embodiment, one or more coefficients of the target ALF are signaled or parsed from the video bitstream.

[0140] In one embodiment, the one or more interpolation filters correspond to one or more upsampling filters, one or more downsampling filters, or both.

[0141] In one embodiment, the first color sample corresponds to a luma sample and the second color sample corresponds to a chroma sample, the first color sample corresponds to a chroma sample and the second color sample corresponds to a luma sample, or the first color sample and the second color sample correspond to two different chroma components.

Brief Description of the Drawings

[0142] Figure 1A An adaptive inter / intra video coding system including loop processing is exemplarily presented.

[0143] Figure 1B Shown Figure 1A The corresponding decoder of the encoder in .

[0144] Figure 2 Shown are the ALF filter shapes for chroma (left) and luma (right) components.

[0145] Figure 3A -D shows the g v (3A), g h (3B), g d1 (3C) and g d2 (3D) Subsampled Laplacian computation.

[0146] Figure 4A Shows the placement of the CC-ALF relative to other loop filters.

[0147] Figure 4B Shows a diamond filter applied to chroma samples.

[0148] Figure 5 Shows an example of the spatial part of a convolutional filter.

[0149] Figure 6 An example of the reference region and the extended region used to derive the filter coefficients is shown.

[0150] Figure 7 The four gradient modes of the Gradient Linear Model (GLM) are shown.

[0151] Figure 8 Examples of the unscaled filter footprint (left) and scaled footprint (right) are shown when the scaling factor is equal to 2.

[0152] Figure 9 A flowchart of an exemplary video coding and decoding system is shown according to an embodiment of the present invention. The system uses a new type of input tap to perform adaptive loop filtering (ALF). [Specific implementation method]

[0153] It will be readily understood that the components of the present invention, as described and illustrated in the figures, may be arranged and designed in a variety of different configurations. Accordingly, the following more detailed description of embodiments of the present systems and methods, as illustrated in the figures, is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References throughout this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic associated with that embodiment may be included in at least one embodiment of the present invention. Therefore, the appearance of the terms "in one embodiment" or "in an embodiment" throughout this specification does not necessarily refer to the same embodiment.

[0154] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. However, one skilled in the relevant art will recognize that the present invention may be practiced without one or more of the specific details, or with other methods, components, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the invention. The embodiments of the present invention will be best understood by reference to the accompanying drawings, in which like parts are numbered throughout. The following description is by way of example only and illustrates only certain selected apparatus and method embodiments consistent with the invention claimed herein.

[0155] To further improve the codec performance of systems using adaptive loop filters (ALFs), new types of input taps are disclosed as follows.

[0156] Model-based tap-based ALF

[0157] In general, the ALF reconstruction process can be expressed as:

[0158]

[0159] Where R(x,y) is the sample value before ALF filtering, is the sample value after ALF filtering, c i is the i-th filter coefficient, n i is the ith filter tap input. Specifically, n i is the clipped neighbor difference, the clipped correction value from another filter, or the clipped correction value from another loop filter stage. In the present invention, an additional tap is introduced, and the new tap input, n i It is generated from some model prediction rather than from existing sample values, where the model can be an interpolation process, a filtering process like ALF, or a prediction process like Convolutional Cross Component Model (CCCM). The above equation shows the general form of ALF. ALF-like (or ALF-type) filtering process refers to any ALF filtering process with a filtering process as shown in the above equation. However, the footprint can have any shape and is not limited to Figure 2 In addition, the number and type of taps are not limited to those disclosed in various versions of ECM. On the other hand, the prediction process similar to CCCM (or CCCM type) can be any convolution filter and is not limited to Figure 5 The CCCM type of prediction process may use Figure 5 More or fewer input samples are shown in .

[0160] Generate tapped ALF from interpolated samples

[0161] In one embodiment, given a scaling factor and a set of interpolation filter coefficients, sample values are generated for a component (Y, Cb, or Cr) at non-integer positions in a region, and the sample values in the upsampled / downsampled regions are used to derive the ALF tap input n for the same component. i .

[0162] Example 1. Derivation of luma ALF tap input using upsampled / downsampled luma.

[0163] Example 2. Derivation of chroma ALF tap input using upsampled / downsampled Cb / Cr.

[0164] In another embodiment, the ALF tap input n of another component is derived using the sample values in the upsampled / downsampled region. i .

[0165] Example 3. Derivation of luma ALF tap input using upsampled / downsampled Cb / Cr.

[0166] Example 4. Deriving chroma ALF or cross-component ALF (CCALF) tap inputs using upsampled / downsampled luma.

[0167] Example 5. Using upsampled / downsampled Cb / Cr to derive chroma ALF tap inputs for other color components Cr / Cb.

[0168] In the above embodiments, the scaling factor and the coefficients of the interpolation filter can be explicitly signaled or implicitly determined. The scaling factor decision can be related to the chroma subsampling format. For example, if the chroma subsampling format is 420, the horizontal and vertical scaling factors are both set to 2 for the chroma components. As another example, if the chroma subsampling format is 422, the horizontal scaling factor is set to 2 and the vertical scaling factor is set to 1 for the chroma components.

[0169] In the above embodiments, either an unscaled filter footprint or a scaled filter footprint is used. For example, if a 3x3 diamond shape is used and the scaling factor is set to 2, Figure 8 Two different options for filter footprints are shown, with the unscaled filter footprint 810 shown on the left and the scaled footprint 820 shown on the right.

[0170] Generate tapped ALF from another ALF filter

[0171] In ECM ALF, the fixed filtering result is used to derive the ALF tap input n i However, fixed filters are trained offline and may sometimes be inappropriate for the current codec frame. Therefore, it is proposed to replace or assist fixed filters with filters from historical adaptive parameter sets (APS).

[0172] In one embodiment, if there is a historical APS, the fixed filter can be replaced with the filter in the historical APS. That is, some ALF tap input n i Instead of being associated with fixed filters, those ALF tap inputs are derived using filters from the historical APS. In another embodiment, if a historical APS exists, additional taps can be introduced into the ALF, where the additional tap inputs are derived using filters from the historical APS. That is, some ALF tap inputs n i Still associated with a fixed filter, the additional ALF tap inputs are derived using the filters from the historical APS.

[0173] Generate tapped ALF from cross-component model (CCM)

[0174] In one embodiment, to perform a chroma ALF filtering process on chroma samples with a cross-component model applied from luma samples to chroma samples, using signaling coefficients, the following steps may be applied:

[0175] Coefficients of a luma-to-chroma cross-component model are derived using reconstructed luma and chroma samples of neighboring reference regions and / or the current block.

[0176] The chroma samples of the current block are derived by applying a luma-to-chroma cross-component model to the luma samples of the current block.

[0177] The chroma filtering process is performed using the chroma samples and the chroma samples from the cross-component model through the signaling coefficients.

[0178] In the above embodiment, a chroma-to-luma cross-component model may also be applied, i.e., in order to perform a luma ALF or CCALF filtering process on luma samples with a cross-component model applied from chroma samples to luma samples, using signaling coefficients, the following steps may be applied:

[0179] Coefficients for a chroma-to-luminance cross-component model are derived using reconstructed chroma and luma samples of neighboring reference regions and / or the current block.

[0180] Derives the luma samples of the current block by applying the chroma-to-luma cross-component model to the chroma samples of the current block.

[0181] The ALF luminance or CCALF filtering process is performed by signaling coefficients with luma samples and luma samples from a cross-component model.

[0182] In the above-described embodiments, the reconstructed luma and chroma samples of the neighboring reference region used to derive the cross-component model coefficients may be samples before the ALF filtering process is applied or samples after the ALF filtering process is applied.

[0183] In the above-described embodiments, the reconstructed luma or chroma samples of the current block used to derive samples from the cross-component model may be samples before the ALF filtering process is applied or samples after the ALF filtering process is applied.

[0184] In the above embodiments, samples from the cross-component model may be used in the filtering process independently and separately from the chroma ALF, luma ALF and / or CCALF filtering processes.

[0185] An example is shown below:

[0186] Luma-to-chroma cross-component model coefficients are derived using reconstructed luma and chroma samples of neighboring reference regions and / or the current block.

[0187] The chroma samples of the current block are derived by applying a luma-to-chroma cross-component model to the luma samples of the current block.

[0188] The chroma filtering process is performed by signaling coefficients with chroma samples from the cross-component model.

[0189] In the above-described embodiments, samples from the cross-component model may be used as reconstructed samples after applying the ALF filtering process.

[0190] In one embodiment, multiple cross-component models can be derived by classifying the reference region into multiple regions, and a cross-component model is derived for each reference region. Then, when applying the cross-component model to the samples of the current block, the corresponding cross-component model is applied according to the category:

[0191] Classification by ALF APS classification or ALF fixed filter classification

[0192] Sort by location

[0193] An example is shown below:

[0194] The neighboring reference area is classified into a plurality of areas.

[0195] Luma-to-chroma cross-component model coefficients are derived using the reconstructed luma and chroma samples of adjacent reference regions, and a cross-component model is derived for each reference region.

[0196] The chroma samples of the current block are derived by applying the luma-to-chroma cross-component model to the luma samples of the current block. Then, for each chroma sample, the luma-to-chroma cross-component model is selected based on the class of the chroma sample or the class of the corresponding luma sample.

[0197] The chroma filtering process is performed by signaling coefficients with chroma samples and chroma samples from the cross-component model.

[0198] Constraints of Multi-Source ALF

[0199] In the exploration of ECM, the ALF process utilizes samples from different codec stages, such as residual samples, prediction samples, co-located reference samples, samples before the deblocking filter, samples before the ALF, and samples filtered by a fixed filter set. In this paper, several constraints are proposed to reduce the buffer usage of this multi-source ALF design.

[0200] Assume that there are N sources available for ALF, denoted as S1, S2, S3, ..., S NIn one embodiment, the sources are selected in the APS (e.g., at the APS level, filter set level, or filter level), and the total number of sources for all samples used in a block, region, slice, tile, sub-image, image, or sequence cannot exceed a number K, where K is an integer less than N.

[0201] An example according to this embodiment is shown below.

[0202] If there are 4 available sources (N=4) and a maximum of 2 sources are allowed (K=2), if the APS, filter set or filter of selection sources S1 and S3 has been selected at the block, region, slice, tile, sub-image, image or sequence level, the APS, filter set or filter of other selection sources S2 or S4 is prohibited from being used for the current block, region, slice, tile, sub-image, image or sequence.

[0203] Another example according to this embodiment is as follows.

[0204] If there are 4 available sources (N=4) and a maximum of 2 sources are allowed (K=2), if APSs, filter sets or filters of selection sources S1 and S3 have been selected at the block, region, slice, tile, sub-image, image or sequence level, then APSs, filter sets or filters of other selection sources S2 or S4 are still allowed to be used for the current block, region, slice, tile, sub-image, image or sequence. However, when applying these APSs, filter sets or filters to samples, pseudo sources will be used instead of the actual sources. For example, if the ALF tap input is generated by using residual samples, but the residual samples are prohibited (one of sources S2 or S4), the tap input will be input to zero, a predetermined constant value or a value derived by replacing the residual samples with pre-ALF samples or other samples from the allowed sources.

[0205] In another embodiment, one source is selected at the block, region, slice, tile, sub-image, image, or sequence level, and the other source is selected in the APS (i.e., at the APS level, filter set level, or filter level). For the first source selection at the block, region, slice, tile, sub-image, image, or sequence level, up to K sources may be selected, where K is an integer less than N.

[0206] Another example according to this embodiment is as follows.

[0207] If there are 4 available sources (N=4) and a maximum of 2 sources are allowed (K=2), if sources S1 and S3 are selected at the block, region, slice, tile, sub-image, image or sequence level, the APS, filter set and filters of source S2 or S4 are prohibited from being selected for the block, region, slice, tile, sub-image, image or sequence.

[0208] Another example according to this embodiment is as follows.

[0209] If there are 4 available sources (N=4) and a maximum of 2 sources (K=2) are allowed, if sources S1 and S3 are selected at the block, region, slice, tile, sub-image, image, or sequence level, then it is still allowed to select an APS, filter set, or filter from source S2 or S4 for the current block, region, slice, tile, sub-image, image, or sequence. However, when applying these APSs, filter sets, or filters to samples, pseudo sources will be used instead of the actual sources.

[0210] In another embodiment, source selection is at the block, region, slice, tile, sub-image, image, or sequence level, and there is no source selection in the APS (i.e., at the APS level, filter set level, or filter level). For source selection, up to K sources may be selected, where K is an integer less than N.

[0211] Another example according to this embodiment is as follows.

[0212] When there are 4 sources available (N=4) and a maximum of 2 sources are allowed (K=2), a maximum of 2 sources are selected at the block, region, slice, tile, sub-image, picture or sequence level and are used to indicate how to use the information signaled in the adaptation parameter set (APS). Specifically, for the current slice, if the "pre-adaptive loop filtering samples" and "residual samples" sources are selected, the filter coefficients signaled in the selected APS will be multiplied by certain values derived from the "pre-adaptive loop filtering samples" and "residual samples" to filter the samples in the current slice; and for another slice, if the "pre-adaptive loop filtering samples" and "collocated reference samples" sources are selected and the same APS is used, the same filter coefficients will be multiplied by certain values derived from the "pre-adaptive loop filtering samples" and "collocated reference samples" to filter the samples.

[0213] The above-mentioned proposed methods can be implemented in an encoder and / or a decoder. For example, the proposed methods can be implemented in a loop filter module of an encoder and / or a loop filter module of a decoder. The adaptive loop filter (ALF) process of the new type of input tap described above can be implemented on the encoder side or the decoder side. For example, any of the proposed methods can be implemented in an inter-frame and / or intra-frame prediction module (e.g., Figure 1A and Figure 1B Any of the proposed methods can also be implemented as a circuit coupled to an inter-frame / intra-frame codec module of a decoder or encoder. However, the decoder or encoder can also use additional processing units to implement the required processing. Although the loop filter unit (e.g., Figure 1A and Figure 1BThe units 130 in FIG. 1 are shown as separate processing units, but they may correspond to executable software or firmware code stored on a medium, such as a hard disk or flash memory, for a central processing unit (CPU) or a programmable device (e.g., a digital signal processor (DSP) or a field programmable gate array (FPGA)).

[0214] Figure 9 A flowchart of an exemplary video codec system using a novel input tap type for adaptive loop filtering (ALF) according to an embodiment of the present invention is provided. The steps shown in the flowchart can be executed as program code on an encoder side on one or more processors (e.g., one or more CPUs). The steps shown in the flowchart can also be implemented in hardware, such as by arranging one or more electronic devices or processors to perform the steps in the flowchart. According to the method, in step 910, input data associated with a current block is received, including a first color block and a second color block, wherein the first color block includes first color samples and the second color block includes second color samples. In step 920, one or more target second color samples are derived based on a cross component model (CCM) applied to one or more CCM input first color samples, or the one or more target second color samples are derived at one or more non-integer positions by applying one or more interpolation filters to one or more interpolated input first color samples or one or more interpolated input second color samples. In step 930, one or more filtered second color samples are generated by applying a target adaptive loop filter (ALF), wherein the ALF uses filtered input samples including the one or more filtered input second color samples and the one or more target second color samples.

[0215] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. A skilled person may modify each step, rearrange steps, split steps, or combine steps to practice the present invention without departing from the spirit of the present invention. In this disclosure, specific syntax and semantics are used to illustrate examples of implementing the present invention. A skilled person may practice the present invention by substituting equivalent syntax and semantics without departing from the spirit of the present invention.

[0216] The above description is intended to enable those of ordinary skill to practice the present invention in the context of specific applications and their requirements. Various modifications of the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but rather to be given the widest scope within the consistent scope of the principles and novel features disclosed herein. In the above detailed description, various specific details are shown to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention can be put into practice.

[0217] The embodiments of the present invention described above can be implemented in various hardware, software code, or a combination of both. For example, one embodiment of the present invention can be one or more circuits integrated into a video compression chip, or program code integrated into video compression software, to perform the processing described herein. One embodiment of the present invention can also be program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The invention may also involve multiple functions performed by a computer processor, a digital signal processor, a microprocessor, or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks by executing machine-readable software code or firmware code that defines the implementation methods of the present invention. The software code or firmware code can be developed in different programming languages and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles, and languages for the software code, as well as other methods of configuring the code to perform tasks consistent with the present invention, do not depart from the spirit and scope of the present invention.

[0218] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalence of the claims are intended to be embraced within their scope.

Claims

1. A method for processing a color image, the method comprising: Receive input data related to a current block, including a first color block and a second color block, wherein the first color block includes a first color sample and the second color block includes a second color sample; deriving one or more target second color samples based on a cross-component model (CCM) applied to one or more CCM input first color samples, or deriving the one or more target second color samples at one or more non-integer positions by applying one or more interpolation filters to one or more interpolated input first color samples or one or more interpolated input second color samples; One or more filtered second color samples are generated by applying a target adaptive loop filter (ALF) using filtered input samples including one or more filtered input second color samples and the one or more target second color samples.

2. The method of claim 1, wherein the cross-component model corresponds to an ALF type filtering process. 3 . The method of claim 2 , wherein coefficients associated with the cross-component model are derived using reconstructed first color samples and reconstructed second color samples of a neighboring reference region and / or the current block. 4 . The method of claim 3 , wherein the reconstructed first color sample and the reconstructed second color sample of the neighboring reference area correspond to reconstructed samples before or after an ALF filtering process. The method of claim 3 , wherein the neighboring reference region is classified into a plurality of regions, and a cross-component model is derived for each of the plurality of regions. 6 . The method of claim 5 , wherein the one or more target second color samples are derived by applying the one cross-component model according to a category associated with the plurality of regions.

7. The method of claim 2, wherein the one or more CCM input first color samples correspond to reconstructed first color samples before or after an ALF filtering process.

8. The method of claim 1, wherein the one or more target second color samples are independently used for the target ALF.

9. The method of claim 8, wherein the target ALF is separate from a luma ALF, a chroma ALF, or a cross-component ALF (CCALF) filtering process.

10. The method of claim 1, wherein the cross-component model corresponds to a convolutional cross-component model (CCCM) type filtering process.

11. The method of claim 1, wherein the one or more target second color samples are used as reconstructed samples after applying an ALF filtering process.

12. The method of claim 1, wherein the one or more coefficients of the target ALF are signaled or parsed from a video bitstream.

13. The method of claim 1, wherein the one or more interpolation filters correspond to one or more upsampling filters, one or more downsampling filters, or both.

14. The method of claim 1 , wherein the first color sample corresponds to a luma sample and the second color sample corresponds to a chroma sample, the first color sample corresponds to a chroma sample and the second color sample corresponds to a luma sample, or the first color sample and the second color sample correspond to two different chroma components.

15. An apparatus for processing encoded and decoded video, the apparatus comprising one or more electronic devices or processors configured to: Receive input data related to a current block, including a first color block and a second color block, wherein the first color block includes a first color sample and the second color block includes a second color sample; deriving one or more target second color samples based on a cross-component model (CCM) applied to one or more CCM input first color samples, or deriving the one or more target second color samples at one or more non-integer positions by applying one or more interpolation filters to one or more interpolated input first color samples or one or more interpolated input second color samples; One or more filtered second color samples are generated by applying a target adaptive loop filter (ALF) using filtered input samples including the one or more filtered input second color samples and the one or more target second color samples.

Images