High-grained decoder-side cross-component loop filter

Abstract

According to an exemplary embodiment of the present invention, as shown in Figure 13, a method and apparatus are provided for performing at least: obtaining at least one convolutional cross-component model filter on a set of samples, wherein the set of samples is reconstructed samples of two channels of an image; applying at least one filter to the set of reconstructed samples of a first channel; applying at least one convolutional cross-component model filter to the set of reconstructed samples of a first channel using the output of at least one filter as input; applying at least one filter to the output of the at least one convolutional cross-component model filter; and applying a cross-component filter to the set of reconstructed samples of a first channel, or applying a cross-component filter to the output of at least one filter, in order to obtain a correction to the output of at least one filter, using the output of at least one convolutional cross-component model filter as input.

Description

[Technical Field]

[0001] The teachings by exemplary embodiments of the present invention relate generally to video coding and decoding, and more specifically to improving chroma construction. [Background technology]

[0002] This section is intended to provide background or context to the invention described in the claims. The descriptions herein may include concepts that can be pursued, but are not necessarily previously invented or pursued. Therefore, unless otherwise indicated herein, the contents of this section are not prior art to the description and claims of this application, nor will they be deemed prior art by being included in this section.

[0003] Certain abbreviations found in descriptions and / or drawings are defined as follows: ALF Adaptive Loop Filter AMVR Adaptive Motion Vector Resolution CC Cross-Component CC-ALF Cross-Component Adaptive Loop Filter CCLM Intersecting Component Linear Model Intra Prediction CTU Encoding Tree Unit CU Central Unit ISM Industry, Science, and Medicine ISP Intra-subpartition LM Linear Model LMS (Least Squares Mean) MRL (Multiple Reference Lines) MMLM Multi-Model LM MVD motion vector difference VVC Multipurpose Video Codec WP weighted prediction

[0004] A brief explanation of the development process so far Block-based processing is widely used in video coding to provide a good balance between coding efficiency and computational complexity. The intra-block copy tool is known to be able to generate a prediction for the current block.

[0005] Exemplary embodiments of the present invention propose improved operations for model monitoring procedures such as beam prediction. SUMMARY OF THE INVENTION

[0006] This section includes, but is not limited to, examples of possible implementations.

[0007] In another exemplary aspect of the present invention, a device such as a user equipment side device includes at least one processor and at least one non-transitory memory that stores instructions which, when executed by the at least one processor, cause the device to obtain at least one convolutional cross-component model filter for a set of samples, where the set of samples is the reconstructed samples of two channels of an image, apply at least one filter to the set of reconstructed samples of the first channel, use the output of the at least one filter as an input to apply the at least one convolutional cross-component model filter to the set of reconstructed samples of the first channel, apply at least one filter to the output of the at least one convolutional cross-component model filter, and use the output of the at least one convolutional cross-component model filter as an input to apply a cross-component filter to the set of reconstructed samples of the first channel or apply a cross-component filter to the output of the at least one filter for the set of reconstructed samples of the first channel to obtain a correction for the output of the at least one filter.

[0008] In another exemplary aspect of the present invention, for a set of samples, obtaining at least one convolutional cross-component model filter, wherein the set of samples is the reconstructed samples of two channels of an image, obtaining, applying at least one filter to the set of reconstructed samples of the first channel, using the output of the at least one filter as an input to apply at least one convolutional cross-component model filter to the set of reconstructed samples of the first channel, applying at least one filter to the output of the at least one convolutional cross-component model filter, and using the output of the at least one convolutional cross-component model filter as an input to apply a cross-component filter to the set of reconstructed samples of the first channel or apply a cross-component filter to the output of the at least one filter for the set of reconstructed samples of the first channel to obtain a correction for the output of the at least one filter.

[0009] A further exemplary embodiment is an apparatus and a method comprising the apparatus and method of the preceding paragraph, wherein the output of the at least one convolutional cross-component model filter may be used as an input to add the output of the cross-component filter to the output of the at least one filter, the convolutional cross-component model filter is derived and applied to blocks that are not determined by coding or prediction partitioning, the blocks have a square or rectangular shape, the blocks may overlap or be separate, and the use of the convolutional cross-component model per block is determined without additional signaling.

[0010] A non-transitory computer-readable medium storing program code, the program code being executed by at least one processor to perform at least the method described in the above paragraph.

[0011] In yet another exemplary aspect of the present invention, there is a device comprising means for adding the output of a cross-component filter to the output of at least one filter, using the output of at least one convolutional cross-component model filter as an input, wherein the convolutional cross-component model filter is derived and applied to blocks not determined by coding or predictive partitions, the blocks having a square or rectangular shape, the blocks may overlap or be separate, and the use of a convolutional cross-component model for each block is determined without additional signaling.

[0012] According to the exemplary embodiments described in the paragraphs above, the means for adding, deriving, applying, and determining comprises at least a network interface and computer program code stored in a computer-readable medium and executed by at least one processor.

[0013] The communication system comprises a network-side device and a user-side device that perform the operations described above.

[0014] The above and other aspects, features, and advantages of various embodiments of this disclosure will be better understood by referring to the following detailed description and accompanying drawings. In the drawings, the same reference numerals are used to indicate identical or equivalent elements. The drawings are provided to facilitate a better understanding of the embodiments of this disclosure and are not necessarily drawn to actual scale. [Brief explanation of the drawing]

[0015] [Figure 1A] This figure shows the sample locations used for the derivation of α and β. [Figure 1B] This figure shows the derivation of the chroma prediction mode from the lumen mode in Table 1 when cclm is enabled. [Figure 1C] This figure shows the unified binary table for the chroma prediction mode in Table 2. [Figure 2]This figure shows the classification of lumar samples into two classes used to derive two sets, α and β, with the sample region (upper) and the spatial region (lower). [Figure 3] This figure shows the sample positions used for deriving the CCCM filter when six reference lines are used. [Figure 4] The diagram shows, from left to right, three taps arranged vertically, three taps arranged horizontally, five taps arranged in a cross shape, and 25 taps arranged in a diamond shape. [Figure 5] This figure shows an example of four reference lines adjacent to a prediction block. [Figure 6] This figure shows the matrix-weighted intra-prediction process. [Figure 7] This figure shows the upper and left-adjacent blocks used in the CIIP weight derivation. [Figure 8] This figure shows the shape of the ALF filter (Chroma: 5x5 diamond, Luma: 7x7 diamond). [Figure 9A] This figure shows the subsampled Laplacian calculation. [Figure 9B] This is a table showing the mapping of calculated gradients and transformations for a single block. [Figure 10] (a) The arrangement of the CC-ALF relative to other loop filters, and (b) the diamond-shaped filter. [Figure 11] This figure shows an improved ALF pipeline using CCCM-based chromatic update. [Figure 12] This is a block diagram of one possible, non-limiting, exemplary system in which exemplary embodiments may be implemented. [Figure 13] This figure shows a method according to an exemplary embodiment of the present invention, which can be performed by the apparatus. [Modes for carrying out the invention]

[0016] In exemplary embodiments of the present invention, at least methods and apparatus are proposed for improving chroma construction for video encoding and decoding.

[0017] Hybrid video codecs, such as ITU-T H.263, H.264 / AVC, and HEVC, can encode video information in two stages. First, the pixel values ​​of a certain picture area (or "block") are predicted, for example, by motion compensation means (finding and indicating an area in one of the previously encoded video frames that closely corresponds to the block being encoded) or by spatial means (using the pixel values ​​around the block to be encoded in a specified manner). In this first stage, predictive coding can be applied, for example, as so-called sample prediction and / or so-called syntax prediction.

[0018] This sample prediction predicts the pixel values ​​or sample values ​​of a given picture area or "block." These pixel values ​​or sample values ​​can be predicted using, for example, one or more motion compensation mechanisms or intra-prediction mechanisms.

[0019] Motion compensation mechanisms (sometimes called interpretation, temporal prediction, or motion-corrected temporal prediction, or motion-corrected prediction or MCP) involve finding and indicating an area in one of the previously encoded video frames that closely corresponds to the block being encoded. Interpretation can reduce temporal redundancy.

[0020] Intra prediction can predict pixel values ​​or sample values ​​using spatial mechanisms. Intra prediction involves finding and demonstrating spatial domain relationships. It leverages the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transformation domains, i.e., it can predict sample values ​​or transformation coefficients. Intra prediction can be used in intra coding where interpretation is not typically applied.

[0021] Syntax prediction, sometimes called parameter prediction, predicts syntax elements, and / or syntax element values ​​and / or variables derived from syntax elements, based on previously encoded (decoded) syntax elements and / or previously derived variables. A non-restrictive example of syntax prediction will be provided later.

[0022] Motion vector prediction allows for the differential encoding of motion vectors, such as motion vectors for inter-prediction and / or interview prediction, relative to the predicted motion vector of a particular block. In many video codecs, the predicted motion vector is generated in a predetermined manner, for example, by calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another method for creating motion vector predictions, sometimes called Advanced Motion Vector Prediction (AMVP), generates a list of candidate predictions from adjacent blocks and / or blocks at the same location as the temporal reference picture, and signals the selected candidates as motion vector predictors. In addition to predicting motion vector values, it is possible to predict the reference index of a previously encoded / decoded picture. The reference index is usually predicted from adjacent blocks and / or blocks at the same location as the temporal reference picture. Differential encoding of motion vectors is usually prohibited across slice boundaries.

[0023] For example, it is possible to predict the block partitioning from CTU to CU, and then to PU.

[0024] Filter parameter prediction can predict filtering parameters, such as filtering parameters for sample adaptation offsets.

[0025] Prediction methods that use image information from previously encoded images are sometimes called interprediction methods, and these methods are sometimes called temporal prediction and motion compensation. Prediction methods that use image information from within the same image are sometimes called intraprediction methods.

[0026] In the second stage, the prediction error, i.e., the difference between the predicted pixel block and the original pixel block, is encoded. This can be done by transforming the difference in pixel values ​​using a specified transformation (e.g., the discrete cosine transform (DCT) or a variation thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By changing the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size at the transmission bitrate).

[0027] In many video codecs, including H.264 / AVC and HEVC, motion information is indicated by motion vectors associated with each motion-compensated image block. Each of these motion vectors represents the displacement between the image block of the picture being encoded (by the encoder) or decoded (by the decoder) and one of the previously encoded or decoded prediction source blocks of the image (or picture). In H.264 / AVC and HEVC, as with many other video compression standards, a single picture is divided into multiple rectangular meshes, each of which is indicated for interpretation by one similar block from the reference picture. The position of the prediction block is encoded as a motion vector that indicates the position of the prediction block relative to the encoded block.

[0028] The Multipurpose Video Codec (VVC) under development includes the following new encoding tools (detailed explanations will be added later to the final patent draft as needed): ●Intra prediction: - 67 intra modes with wide-angle mode extension - Block size and mode-dependent 4-tap interpolation filter - Location-dependent intra-predictive combination (PDPC) - Cross-Component Linear Model Intra-Prediction (CCLM) - Multiple reference line intra prediction - Intra subpartition - Weighted intra prediction using matrix multiplication ●Picture-to-picture prediction: - Block motion copying using spatial, temporal, history-based, and pairwise average merge candidates - Affine motion interface prediction - Prediction of temporal motion vectors based on subblocks - Adaptive motion vector resolution - Motion compression based on 8x8 blocks for temporal motion prediction - High-precision (1 / 16 pixel) motion vector storage and motion compensation using an 8-tap interpolation filter for the lumens component and a 4-tap interpolation filter for the chromens component. - Triangular partition - Combined intra-prediction and inter-prediction - Merging using MVD (MMVD) - Symmetric MVD coding - Bidirectional optical flow - Refinement of decoder-side motion vector - Bidirectional prediction using CU-level weights ●Transformation, quantization, and coefficient coding: - Multiple linear transformation selection using DCT2, DST7, and DCT8 - Secondary conversion for the low-frequency zone - Subblock transformation for predicted inter-residuals - Dependent quantization using maximum QP increased from 51 to 63 - Transformation coefficient coding using sign data hiding - Conversion skip residual coding ● Entropic coding: - Arithmetic coding engine using adaptive double-window probability updates ● In-loop filter: - In-loop reshaping - Deblocking filter using a more powerful and longer filter - Sample-adaptive offset - Adaptive loop filter ●Screen content encoding: - Current picture referencing using reference area restrictions ● 360-degree video encoding: - Horizontal wrap-around motion compensation ● High-level syntax and parallel processing: - Reference picture management using direct reference picture list signaling - Tile group containing rectangular tile group

[0029] Partitioning in VVC In VVC, each picture is divided into coding tree units (CTUs) similar to those in HEVC. Pictures can also be divided into slices, tiles, bricks, and subpictures. A quaternary tree structure can be used to further divide CTUs into smaller CUs. Each CU can be further divided using nested multi-type trees, including quadtrees, ternary, and binary partitions.

[0030] There are specific rules for inferring picture boundary partitioning.

[0031] Redundant partition patterns are not permitted in nested multi-type partitioning.

[0032] Cross-Component Linear Model Prediction (CCLM) To reduce redundancy between cross-components, the Cross-Component Linear Model (CCLM) prediction mode is used in VVC, in which the chroma sample is predicted based on the reconstructed lumen sample of the same CU by using a linear model as follows: pred_C(i,j)=α·rec_L '(i,j)+β(3-1) In the above formula, pred C (i,j) represents the predicted chroma sample within CU, and rec L '(i,j) represents a downsampled and reconstructed rumor sample of the same CU.

[0033] The CCLM parameters (α and β) are derived using up to four adjacent chromatic samples and their corresponding downsampled chromatic samples. If the dimensions of the current chromatic block are W × H, then W' and H' are set as follows: - When LM mode is applied, W'=W, H'=H - When LM-A mode is applied, W'=W+H - When LM-L mode is applied, H'=H+W

[0034] The upper adjacent positions are represented as S[0,-1]...S[W'-1,-1], and the left adjacent positions are represented as S[-1,0]...S[-1,H'-1]. Then, four samples are selected as follows. - When LM mode is applied and both the upper adjacent sample and the left adjacent sample are available, S[W' / 4,-1], S[3*W' / 4,-1], S[-1,H' / 4], S[-1,3*H' / 4] - When LM-A mode is applied, or when only the upper adjacent sample is available, S[W' / 8,-1], S[3*W' / 8,-1], S[5*W' / 8,-1], S[7*W' / 8,-1] - When LM-L mode is applied, or when only left-neighbor samples are available, use S[-1,H' / 8], S[-1,3*H' / 8], S[-1,5*H' / 8], S[-1,7*H' / 8]

[0035] Downsample the four adjacent lumasamples at the selected position, compare them four times, and find two smaller values x0A and x1A and two larger values x0B and x1B. Their corresponding chroma sample values are represented as y0A, y1A, y0B, and y1B. Then, xA, xB, yA, and yB are derived as follows. -X a =(x 0 A +x 1 A +1)>>1;X b =(x 0 B +x 1 B +1)>>1;Y a =(y 0 A +y 1 A +1)>>1;Y b =(y 0 B +y[[ID=---]] 1 B +1)>>1 (3-1)

[0036] Finally, the linear model parameters α and β are obtained according to the following formula.

Equation

[0037] Figure 1 shows an example of the positions of the left and upper samples and the samples of the current block involved in the CCLM mode.

[0038] The division operation for calculating the parameter α is performed using a lookup table. To reduce the memory required to store this table, the diff value (the difference between the maximum and minimum values) and the parameter α are expressed in exponential notation. For example, diff is approximated using a 4-bit mantissa part and an exponent part. Therefore, the table of 1 / diff is reduced to 16 elements for 16 values of the mantissa as follows. DivTable[]={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} (3-4)

[0039] This has the advantage of reducing the complexity of the calculations and the amount of memory required to store the necessary tables.

[0040] Alternatively, in addition to being able to calculate linear model coefficients together using the upper and left templates, these templates can also be used in two other LM modes called LM_A mode and LM_L mode.

[0041] In LM_A mode, the linear model coefficients are calculated using only the upper template. To obtain more samples, the upper template is extended to (W+H). In LM_L mode, the linear model coefficients are calculated using only the left template. To obtain more samples, the left template is extended to (H+W).

[0042] For non-square blocks, the top template is extended to W+W, and the left template is extended to H+H.

[0043] To align the chroma sample positions for a 4:2:0 video sequence, two types of downsampling filters are applied to the chroma samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions. The selection of the downsampling filter is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "Type 0" and "Type 2" content, respectively.

[0044]

number

[0045] Note that when the upper reference line is on the CTU boundary, only one lumern line (a common line buffer in intra-prediction) is used to create a downsampled lumern sample.

[0046] This parameter calculation is performed as part of the decoding process, not solely as an encoder search operation. Consequently, no syntax is used to communicate the α and β values ​​to the decoder.

[0047] For chroma intra-mode coding, a total of eight intra-modes are allowed. These modes include five traditional intra-modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). The chroma mode signaling and derivation processes are shown in Table 1 of Figure 1B below. Chroma mode coding directly depends on the intra-prediction mode of the corresponding rumor block. Since separate block partitioning structures for rumor and chroma components are made available in I-slices, one chroma block may correspond to multiple rumor blocks. Therefore, for chroma DM modes, the intra-prediction mode of the corresponding rumor block covering the central position of the current chroma block is directly inherited.

[0048] In Table 2 of Figure 1C, the first binary (bin) indicates whether it is normal mode (0) or LM mode (1). If it is LM mode, the next binary indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next binary indicates whether it is LM_L (0) or LM_A (1). In this case, when sps_cclm_enabled_flag is 0, the first binary in the corresponding intra_chroma_pred_mode binary table may be discarded before entropy coding. Or, in other words, the first binary is inferred to be 0 and therefore not coded. This single binary table is used for both cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two binaries in Table 2 are context-coded using their own context model, while the other binaries are bypass-coded.

[0049] Furthermore, to reduce rumor-chroma latency in dual trees, when a 64x64 rumor-encoded tree node is partitioned with Not Split (ISP is not used for 64x64 CUs) or QT, the chroma CU of a 32x32 / 32x16 chromor-encoded tree node is allowed to use CCLM in the following manner. - If a 32x32 chroma node is not partitioned or not partitioned with QT partitioning, all chroma CUs on the 32x32 node can use CCLM. - If a 32x32 chroma node is partitioned with Horizontal BT, and a 32x16 child node is not partitioned but uses Vertical BT partitioning, then all chroma CUs of the 32x16 chroma node can use CCLM.

[0050] For all other lumar and chromar coding tree partitioning conditions, CCLM is not allowed for chromar CU.

[0051] Multi-model LM (MMLM) The CCLM included in VVC is extended by adding three multi-model LM (MMLM) modes. In each MMLM mode, reconstructed neighbor samples are classified into two classes using a threshold that is the mean of the reconstructed neighbor samples of the rumor. The linear model for each class is derived using the least squares mean (LMS) method. In the CCLM mode as well, the LMS method is used to derive the linear model. Figure 2 shows two rumor-chroma models obtained when the threshold for rumor (Y) is set to 17. Each rumor-chroma model has its own linear model parameters α and β. As can be seen from the figure below, each rumor-chroma model corresponds to a spatial segmentation of the content (i.e., to different objects or textures in the scene).

[0052] Convolutional Cross-Component Model (CCCM) An improved version of cross-component prediction, known as CCCM, derives a lumer-chroma model using a 2D filter kernel. Filter coefficients are derived on the decoder side using a set of reconstructed input data and chroma samples. In deriving the filter coefficients, a reference sample area (consisting of reconstructed lumer and chroma samples) is defined for both lumer and chroma at the same location, as shown in Figure 3, where the commonly used 4:2:0 chroma downsampling is applied. The reference sample area for a given block can be six rows above and to the left, as shown in Figure 3, but any number of reference lines (achievable on both the encoder and decoder) can be used. Generally, the reference samples can include any chroma and lumer samples reconstructed by both the encoder and decoder. Once the reference samples are determined, the filter coefficients can be derived using various types of linear regression tools, such as standard least squares estimation, orthogonal matching tracking, optimized orthogonal matching tracking, ridge regression, or the least absolute condensation selection operator.

[0053] The dimensions of the filter kernel can be arbitrary, such as 1x3 (one-dimensional vertical), 3x1 (one-dimensional horizontal), 3x3, 7x7, etc., and its shape can also be a cross or a rhombus (as shown in Figure 4) or any shape (by selecting only a subset from all possible kernel positions). The following notation is used to refer to samples within the filter kernel, as shown in Figure 4 using the letters N, E, S, W, and C: North (top), East (right), South (bottom), West (left), and Center.

[0054] The overall method for reconstructing chroma samples using a convolution operation between a filter kernel acquired on the decoder side and a set of input data is referred to herein as the Convolutional Cross-Component Model (CCCM). The following steps can be applied to perform CCCM operation. 1. Define a reference area that is in the same position for the lumen component and the chroma component. 2. Downsample lumens to align with the chroma grid (optional). 3. Scan the lumens and chroma samples of the reference area and collect available statistics (such as autocorrelation matrices and cross-correlation vectors) based on the filter shape. 4. Solve the filter coefficients by minimizing the squared error (or any other indicator) based on available parameters (such as the autocorrelation matrix and cross-correlation vector). 5. The predicted chroma block is computed by convolving the downsampled chroma samples with a filter kernel.

[0055] A chroma sample (which may be downsampled) is defined as a 2D array Y(x, y) indexed using the horizontal x-coordinate and vertical y-coordinate. A chroma sample at the same position is defined as a 2D array C(x, y), and the filter kernel (i.e., coefficients) is defined as a 3x3 array F(i, j). At the sample level, the convolution operation between Y and F is defined as follows:

[0056]

number

[0057] When using other data terms such as a nonlinear square root term, the added convolution is as follows:

number

number

number

[0058] Multiple Reference Lines (MRL) Intra-Prediction Multiple Reference Line (MRL) intra-prediction uses more reference lines for intra-prediction. Figure 5 shows an example with four reference lines, illustrating that the sample values ​​for segments A and F are not fetched from reconstructed adjacent samples, but rather are interpolated with the closest samples from segments B and E, respectively. HEVC intra-picture prediction uses the closest reference line (i.e., reference line 0). MRL uses two auxiliary lines (reference line 1 and reference line 3).

[0059] The index of the selected reference line (mrl_idx) can be signaled and used to generate an intra predictor. For reference line idx greater than 0, only the additional reference line modes can be included in the MPM list, and only the MPM index can be signaled without including the remaining modes. The reference line index can be signaled before the intra predictor mode, and if a non-zero reference line index is signaled, the Planar mode can be excluded from the intra predictor mode.

[0060] MRL can be disabled for the first line of the inner block of the CTU, preventing the use of extended reference samples outside the current CTU line. Furthermore, PDPC can be disabled when auxiliary lines are used. For MRL mode, the derivation of the DC value for DC intra-prediction mode for non-zero reference line indices is aligned with that for reference line index 0. MRL requires storage of three adjacent lumen reference lines with the CTU to generate predictions. The Cross-Component Linear Model (CCLM) tool also requires three adjacent lumen reference lines for its downsampling filter. To reduce the decoder's storage requirements, the definition of MRL, which uses the same three lines, is aligned with CLM.

[0061] Intra-subpartition (ISP) Intra-subpartitions (ISPs) divide intra-predicted rumor blocks vertically or horizontally into two or four subpartitions depending on the block size. For example, the minimum block size for an ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided into four subpartitions. It has been found that Mx128 (M≦64) and 128xN (N≦64) ISP blocks can generate potential problems, including 64x64 VDPUs. For example, Mx128 CUs in the case of a single tree would result in Mx128 rumor TB and two corresponding

[0062]

number

[0063] Matrix-weighted intra-prediction (MIP) Matrix-weighted intra-prediction (MIP) is a new intra-prediction technique added to VVC. To predict samples of a rectangular block of width W and height H, Matrix-weighted intra-prediction (MIP) uses two inputs: one line of reconstructed adjacent boundary samples of height H located on the left side of the block, and another line of reconstructed adjacent boundary samples of width W located at the top of the block. If reconstructed samples are unavailable, they are generated in a manner similar to conventional intra-prediction. The generation of the prediction signal is based on three steps: averaging, matrix-vector multiplication, and linear interpolation, as shown in Figure 6.

[0064] Intraprediction in VVC The merge list may include the following candidates: 1) Spatial MVP from spatially neighboring CU, 2) Temporal MVP from coexisting CUs, 3) History-based MVP from FIFO table, 4) Pairwise average MVP (using candidates already included in the list), 5) No music videos.

[0065] The Merge Mode Width Motion Vector Difference (MMVD) signals merge candidates, followed by the MVD and resolution index.

[0066] In symmetric MVD, for bidirectional prediction, the motion information in List 1 is derived from the motion information in List 0.

[0067] In affine prediction, multiple motion vectors are indicated / signaled for different corners of a block, and these are used to derive the motion vectors of subblocks. In affine merging, affine motion information for a block is generated based on the normal direction or affine motion information of adjacent blocks.

[0068] In subblock-based temporal motion vector prediction, the motion vector of the current block's subblocks is predicted from the appropriate subblocks in the reference frame, which are indicated by the motion vectors of the spatially adjacent blocks (if available).

[0069] In adaptive motion vector resolution (AMVR), the accuracy of the MVD is signaled for each CU.

[0070] In bidirectional predictions with CU-level weights, the index indicates the weighted average of the two prediction blocks.

[0071] Bidirectional Optical Flow (BDOF) refines motion vectors in the case of bidirectional prediction. BDOF generates two prediction blocks using the signaled motion vectors. Then, the gradient values ​​of the two prediction blocks are used to calculate motion refinement to minimize the error between them. The final prediction block is refined using the motion refinement and gradient values.

[0072] Bidirectional prediction using CU-level weights (BCW) and weighted predictions (WP) In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or by using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging, allowing for a weighted average of the two prediction signals. P bi-pred =((8-w)*P0+w*P1+4)>>3

[0073] In weighted average bidirectional prediction, five weights w ∈ {-2, 3, 4, 5, 10} are allowed. For each CU predicted bidirectionally, the weights w are determined in one of two ways: 1) For non-merged CUs, the weight indices are signaled after the motion vector difference; 2) For merged CUs, the weight indices are inferred from adjacent blocks based on the merge candidate index. BCW is applied only to CUs with 256 or more lumane samples (i.e., CU width × CU height is 256 or greater). For low-latency pictures, all five weights are used. For non-low-latency pictures, only three weights (w ∈ {3, 4, 5}) are used. - In the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the complexity of the encoder. These algorithms are summarized below. For further details, please refer to the VTM software and documentation JVET-L0646. When combined with AMVR, unequal weighting is conditionally checked for 1-pixel and 4-pixel motion vector accuracy only if the current picture is a low-latency picture. - When combined with affine, affine ME is performed on unequal weights only if and only if the affine mode is selected as the current optimal mode. - In bidirectional prediction, if the two reference pictures are the same, unequal weighting is checked only conditionally. - Depending on the POC distance between the current picture and its reference picture, the encoded QP, and the time level, unequal weights are not searched if certain conditions are met.

[0074] The BCW weight index is encoded using one context-coded bin followed by a bypass-coded bin. The first context-coded bin signals whether equal weights are being used, and if unequal weights are being used, an additional bin is signaled using bypass coding to indicate which unequal weights are being used.

[0075] Weighted Prediction (WP) is a coding tool supported by the H.264 / AVC and HEVC standards for efficiently encoding video content with fading. WP support has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be signaled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. To avoid complicating the VVC decoder design due to the interaction between WP and BCW, when a CU uses WP, the weight index of the BCW is not signaled, and w is estimated to be 4 (i.e., equal weighting is applied). In the case of merged CUs, the weight index is inferred from adjacent blocks based on the merge candidate index. This applies to both normal merge mode and inherited affine merge mode. In constructed affine merge mode, affine motion information is built on motion information from up to 3 blocks. The BCW index of a CU using the constructed affine merge mode is simply set to be equal to the BCW index of the first control point MV.

[0076] In VVC, it is not possible to apply CIIP and BCW to a CU simultaneously. If a CU is encoded in CIIP mode, the BCW index of the current CU is set to 2, for example, equal weight.

[0077] Combined Interpretation and Intrapretation (CIIP) In VVC, when a CU is encoded in merge mode, if the CU contains at least 64 lumens (i.e., the product of the CU width and CU height is 64 or greater) and both the CU width and CU height are less than 128 lumens, an additional flag is signaled to indicate whether the inter / intra prediction (CIIP) mode combined with the current CU is applied. As the name indicates, CIIP prediction combines the inter-prediction signal and the intra-prediction signal. The inter-prediction signal P in CIIP mode inter This is derived using the same interprediction processing applied to the normal merge mode, and the intraprediction signal P intra This is derived according to the normal intra-prediction process in planar mode. The intra-prediction signal and the inter-prediction signal are then combined using a weighted average, and the weight values ​​are calculated as follows, depending on the coding mode of the upper and left adjacent blocks (shown in Figure 7). - Set isIntraTop to 1 if the upper adjacent is available and intra-encoded, otherwise set isIntraTop to 0. - Set isIntraLeft to 1 if the left adjacent is available and intra-encoded, otherwise set isIntraLeft to 0. - If (isIntraLeft + isIntraTop) is equal to 2, then wt is set to 3. - Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, wt is set to 2. - Otherwise, set wt to 1.

[0078] LIC is an inter-prediction technique that models the change in local illumination between the current block and its predicted block as a function of the change between the current block template and the reference block template. The parameters of the function can be expressed by scale α and offset β, forming a linear equation for correcting the change in illumination, namely α*p[x]+β, where p[x] is the reference sample pointed to by the MV at position x of the reference picture. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required except that the LIC flag is signaled in AMVP mode to indicate the use of LIC.

[0079] The local illumination correction proposed in JVET-O0066 is used in a single predictive interCU in ECM with the following modifications: ● Intra-neighbor samples can be used in the derivation of LIC parameters. ●LIC is disabled in blocks with fewer than 32 luma samples. ●In both non-subblock mode and affine mode, the derivation of LIC parameters is performed based on the template block sample corresponding to the current CU, rather than the partial template block sample corresponding to the first upper-left 16x16 unit. ●The sample reference block template is generated by using MC without rounding the block MV to integer pixel precision.

[0080] Handling of Out-of-Boundary (OOB) Samples In bidirectional motion compensation, out-of-bounds (OOB) prediction samples are discarded, and non-OOB predictors are used only if available to generate the final predictor. Specifically, Pos_x i,j and Pos_y i,j This indicates the position of one predicted sample within one current block.

[0081]

number

number

number

number

number

number

number

number

number

number

[0082] Finally, it should be noted that this sample-adaptive bidirectional prediction process applies only to the first prediction unit where the reference block is detected to be at least partially or completely out of bounds. Therefore, the block-level OOB criterion is checked first. If both prediction blocks are not OOB, normal bidirectional prediction is performed.

[0083] In-loop filter VVC has a total of three in-loop filters. In addition to the deblocking filter and SAO (the two loop filters in HEVC), an adaptive loop filter (ALF) is applied. The ALF consists of a lumens ALF, a chromens ALF, and a cross-component ALF (CC-ALF). The ALF filtering process is designed to run the lumens ALF, chromens ALF, and CC-ALF in parallel. The filtering order in VVC is deblocking filter, SAO, and ALF. The SAO in VVC is the same as the SAO in HEVC.

[0084] In VVC, an Adaptive Loop Filter (ALF) with block-based filter adaptation is applied. For the lumens component, one of 25 filters is selected for every 4x4 block based on the direction of the local gradient and activity.

[0085] Two diamond filter shapes (as shown in Figure 8) are used. The 7x7 diamond shape is applied to the lumens component, and the 5x5 diamond shape is applied to the chromatic component.

[0086] For the luma components, each 4x4 block is classified into one of 25 classes. The classification index C is determined by its direction D and the quantization value of its activity.

number

number

number

number

[0087] To reduce the complexity of block classification, a subsampled one-dimensional Laplacian calculation is applied. As shown in Figure 9, the same subsampling locations are used for gradient calculations in all directions.

[0088] Next, the maximum and minimum values ​​of the horizontal and vertical gradients D are set as follows:

number

number

number

number

number

number

number

[0089] Activity value A is calculated as follows:

number

number

[0090] The classification method does not apply to the chroma component within a picture.

[0091] Before filtering each 4x4 lumens block, geometric transformations such as rotation or diagonal and vertical inversion are applied to the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l), depending on the gradient values ​​calculated for that block. This is equivalent to applying these transformations to the samples within the filter support region. The aim is to enhance their similarity by aligning the orientation of different blocks to which ALF is applied.

[0092] Three geometric transformations are introduced, including diagonal, vertical inversion, and rotation. Diagonal: f D (k,l)=f(l,k), c D (k,l)=c(l,k) Vertical inversion: f V (k,l)=f(k,Kl-1)c V (k,l)=c(k,Kl-1), Rotation: f R (k,l)=f(Kl-1,k), c R (k,l)=c(Kl-1,k) In the above equation, K is the size of the filter, 0 ≤ k, and l ≤ K-1 are the coordinates of the coefficients, so the position (0,0) is the upper left corner and the position (K-1,K-1) is the lower right corner. The transformation is applied to the filter coefficients f(K,l) and clipping value c(K,l) according to the gradient values ​​calculated for that block. The relationship between the transformation and the four gradients in the four directions is summarized in the following table.

[0093] If ALF is enabled for CTB on the decoder side, each sample R(i,j) in CU is filtered, and the sample value R'(i,j) is obtained as shown below.

number

number

number

[0094] The clipping function K(x,y)=min(y,max(-y,x)) corresponds to the function Clip3(-y,y,x). The clipping operation introduces nonlinearity and improves the efficiency of ALF by mitigating the influence of neighboring sample values ​​that differ significantly from the current sample value.

[0095] CC-ALF uses lumen sample values ​​to refine each chroma component by applying an adaptive linear filter to the lumen channel and then using the output of this filtering operation. Figure 10(a) provides a system-level diagram of the CC-ALF process relating to the SAO, lumen ALF, and chromen ALF processes.

[0096] Filtering in CC-ALF is achieved by applying a linear diamond filter (Figure 10(b)) to the chroma channel. One filter is used for each chroma channel, and the operation is expressed as follows:

number

[0097] As shown in Figure 10, the lumern filter support is the region that coexists with the current chroma sample after considering the spatial scaling factor between the lumern plane and the chroma plane.

[0098] In the VVC reference software, CC-ALF filter coefficients are calculated by minimizing the mean squared error of each chroma channel relative to the original chroma content. To achieve this, the VTM algorithm uses a coefficient derivation process similar to that used for chroma ALF. Specifically, a correlation matrix is ​​derived, and coefficients are calculated using a Choleski decomposition solver to minimize the mean squared error index. When designing filters, up to eight CC-ALF filters can be designed and sent per picture. The resulting filters are then indicated for every two chroma channels on a CTU basis.

[0099] Adaptive in-loop filters (such as ALF in VVC) perform lumern and chromar filtering in parallel to reduce the mean squared error (MSE) between the reconstruction and the original sample. More specifically, in VVC, ALF first filters both lumern and chromar in parallel, and then applies cross-component filtering (CC-ALF) to further refine the chromar (see Figure 10a). Because the filters are signaled at a coarse spatial granularity, the filters are modified at most at the CTU level.

[0100] In VVC, CC-ALF uses the lumen sample prior to ALF as input, thus ignoring the improvements obtained from lumen ALF. Even if CC-ALF considered the ALF-filtered lumen as input, it would not offer much advantage because CC-ALF uses a Wiener filter (signaled at the CTU level) to improve the chroma, and at this scale, small changes to the input of the Wiener filter derivation do not significantly alter the output (i.e., the filter coefficients are slightly different, but the output is almost the same).

[0101] To fully translate rumor improvements into chroma improvements, a more local (and independent of encoding or prediction partitioning) rumor-to-chroma filter is needed instead of CTU-level filters, but signaling such filters is prohibited due to its high signaling cost.

[0102] To improve the use of the ALF's lumen output for improving chroma reconstruction, exemplary embodiments of the present invention apply filters that can directly convert lumen improvements to chroma improvements before chroma ALF or CC-ALF is applied. These filters need to be derived and applied locally with fine spatial granularity (i.e., a different filter for each 4x4 block, for example). According to exemplary embodiments of the present invention, a convolutional cross-component model (CCCM) is used to map the improved lumen to the improved chroma without additional signaling. Variations of such methods have already been described in [1], where lumen residuals are mapped to chroma corrections using CCCM.

[0103] Before describing in detail the exemplary embodiments disclosed herein, Figure 12 shows a simplified block diagram of various electronic devices suitable for use in carrying out exemplary embodiments of the present invention.

[0104] Figure 12 is a block diagram of one possible, non-limiting, exemplary system in which exemplary embodiments may be implemented. In Figure 12, as shown in Figure 12, the user equipment (UE) 10 is communicating wirelessly with wireless network 1 or network 1. Wireless network 1 or network 1 in Figure 12 may comprise a communication network such as a mobile network, for example, mobile network 1 or first mobile network disclosed herein. References to wireless network 1 in Figure 12 herein can be considered as references to any wireless network disclosed herein. Furthermore, wireless network 1 in Figure 12 may also comprise hardwired functionality if required by the communication network. The UE is a wireless, typically mobile device, capable of accessing the wireless network. For example, the UE may be a mobile phone (or "cellular" phone) and / or a computer with mobile terminal capabilities. For example, the UE or mobile terminal may be a portable, pocket-sized, handheld, computer-integrated, or vehicle-mounted mobile device that performs language signaling and / or data exchange with the RAN.

[0105] UE10 includes one or more processors DP10A, one or more memory MEM10B, and one or more transceivers TRANS10D, which are interconnected via one or more buses. Each of the one or more transceivers TRANS10D includes a receiver and a transmitter. The one or more buses may be an address bus, a data bus, or a control bus, and may include a series of lines on a motherboard, or any interconnection mechanism such as an integrated circuit, optical fiber, or other optical communication equipment. Each of the one or more transceivers TRANS10D can optionally be connected to one or more antennas for communication with NN12 and ISM13. One or more memory MEM10B contains computer program code PROG10C. UE10 communicates with NN12 and / or ISM13 via wireless links 11 or 16.

[0106] NN12 (NR / 5G Node B, Evolutionary NB, or LTE device) is a network node, such as a master or secondary node base station (for NR or LTE Long Term Evolution), that communicates with devices such as ISM13 and UE10 in Figure 12. NN12 enables wireless devices such as UE10 to access the wireless network 1. NN12 includes one or more processors DP12A, one or more memory MEM12B, and one or more transceivers TRANS12D, which are interconnected via one or more buses. According to an exemplary embodiment, these TRANS12D may include X2 and / or Xn interfaces used to perform the exemplary embodiment. Each of the one or more transceivers TRANS12D includes a receiver and a transmitter. One or more transceivers TRANS12D can optionally be connected to one or more antennas to communicate with UE10 via at least link 11. One or more memory MEM12B and computer program code PROG12C, together with one or more processors DP12A, are configured to cause NN12 to perform one or more of the operations described herein. NN12 can communicate with other gNB or eNB or devices such as ISM13, for example, via link 16. Furthermore, links 11, 16, and / or other links can be wired or wireless, or both, and can implement, for example, X2 or Xn interfaces. Additionally, links 11 and / or 16 may be configured through other network devices such as NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF14 devices, as shown in Figure 12, but are not limited to these. NN12 can perform functions of an MME (Mobility Management Entity) or SGW (Service Gateway), such as user plane functions and / or access management functions for LTE, and similar functions for 5G.

[0107] ISM13 may be for WiFi or Bluetooth, or other wireless devices associated with mobility function devices such as AMF or SMF, and further, ISM13 may include base stations such as master or secondary node base stations (e.g., for NR or LTE Long Term Evolution) that communicate with devices such as NR / 5G node B, or potentially evolved NB, and NN12 and / or UE10 and / or wireless network 1. ISM13 includes one or more processors DP13A, one or more memory MEM13B, one or more network interfaces, and one or more transceivers TRANS13D, which are interconnected via one or more buses. According to an exemplary embodiment, these network interfaces of ISM13 may include X2 and / or Xn interfaces used to perform the exemplary embodiment. Each of the one or more transceivers TRANS13D includes a receiver and a transmitter, which are optionally connectable to one or more antennas. One or more memory MEM13B includes computer program code PROG13C. For example, one or more memory MEM13B and computer program code PROG13C, together with one or more processors DP13A, are configured to cause the ISM13 to perform one or more of the operations described herein. The ISM13 can communicate with other mobility function devices and / or eNBs, such as the NN12 and UE10, or any other devices, using, for example, link 11 or link 16, or other links. Link 16, shown in Figure 12, can be used to communicate with the NN12. These links can be wired or wireless, or both, and can implement, for example, X2 or Xn interfaces. Furthermore, as stated above, links 11 and / or link 16 may be configured through other network devices, such as NCE / MME / SGW devices, such as the NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF14 shown in Figure 12, but are not limited to these.

[0108] One or more buses in the device shown in Figure 12 may be address buses, data buses, or control buses, and may include a series of lines on a motherboard, or any interconnection mechanism such as integrated circuits, optical fibers, other optical communication equipment, or wireless channels. For example, one or more transceivers TRANS12D, TRANS13D, and / or TRANS10D may be implemented as a remote radio head (RRH), with the other elements of NN12 located physically separate from the RRH, and these devices may include one or more buses, some of which may be implemented as optical fiber cables, to connect the other elements of NN12 to the RRH.

[0109] Figure 12 shows network nodes such as NN12 and ISM13, but it should be noted that any of these nodes can incorporate or be incorporated into eNodeB, eNB, or gNB for LTE, NR, etc., and are still configurable to perform the exemplary embodiment.

[0110] While the descriptions herein indicate that a “cell” performs a function, it should be clear that the gNB and / or user equipment and / or mobility management device forming the cell perform the function. Furthermore, a cell constitutes part of a gNB, and there may be multiple cells in a single gNB.

[0111] Wireless network 1 or any network it may represent may include or may not include NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF14, which may include (NCE) Network Control Element functions, MME (Mobility Management Entity) / SGW (Service Gateway) functions, and / or Service Gateway (SGW), as well as / or MME (Mobility Management Entity) and / or SGW (Service Gateway) functions, as well as / or User Data Management Function (UDM), as well as / or PCF (Policy Control) functions, as well as / or Access and Mobility Management Function (AMF) functions, as well as / or Session Management (SMF) functions, as well as / or Location Management Function (LMF), as well as / or Authentication Server (AUSF) functions, and provide connectivity to further networks such as telephone networks and / or data communication networks (e.g., the Internet), and are configured to perform 5G and / or NR operations in addition to or instead of other standard operations as of the present filing date. The NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF14 can be configured to perform the exemplary actions in any communication technology, including LTE, NR, 5G, and / or communication technologies based on any standards that are in operation or under discussion at the time of this filing. Furthermore, it should be noted that the exemplary actions performed by NN12 and / or ISM13 are also possible with the NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF14.

[0112] The NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF14 includes one or more processors DP14A, one or more memory MEM14B, and one or more network interfaces (N / WI / F), which are interconnected via one or more buses and coupled to link 13 and / or link 16. According to an exemplary embodiment, these network interfaces may include X2 and / or Xn interfaces used to perform the exemplary embodiment. One or more memory MEM14B includes computer program code PROG14C. One or more memory MEM14B and computer program code PROG14C, together with one or more processors DP14A, are configured to cause the NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF14 to perform one or more operations that may be necessary to support the operation according to the exemplary embodiment.

[0113] It should be noted that NN12 and / or ISM13 and / or UE10 can be configured (for example, based on standard implementations) to perform location management function (LMF) functions. The LMF function may be embodied in any of these network devices or any other device associated with these devices. Furthermore, at least the LMFs described later, such as the LMF of MME / SGW / UDM / PCF / AMF / SMF / LMF14 shown in Figure 12, may be located in the same position as UE10, separated from NN12 and / or ISM13 in Figure 12, in order to perform the operations according to the exemplary embodiments disclosed herein.

[0114] Wireless Network 1 can implement network virtualization, which is the process of combining hardware and software network resources and network functions into a virtual network, a single software-based management entity. Network virtualization includes platform virtualization and is often combined with resource virtualization. Network virtualization is classified into external types, which consolidate many networks, or parts of networks, into a single virtual unit, and internal types, which provide network-like functionality to software containers on a single system. It should be noted that the virtualized entities resulting from network virtualization still have technical effects because they are implemented to some extent using hardware such as processors DP10, DP12A, DP13A, and / or DP14A, and memory MEM10B, MEM12B, MEM13B, and / or MEM14B.

[0115] The computer-readable memories MEM12B, MEM13B, and MEM14B may be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The computer-readable memories MEM12B, MEM13B, and MEM14B may also be means for performing storage functions. The processors DP10, DP12A, DP13A, and DP14A may be of any type suitable for the local technical environment and, as non-limiting examples, may include one or more general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), and processors based on multicore processor architectures. The processors DP10, DP12A, DP13A, and DP14A may be means for performing functions such as controlling UE10, NN12, ISM13, and other functions described herein.

[0116] Generally, various embodiments of any of these devices, though not limited to them, may include cellular phones such as smartphones, tablets, personal digital assistants (PDAs) with wireless communication capabilities, portable computers with wireless communication capabilities, image capture devices such as digital cameras with wireless communication capabilities, gaming devices with wireless communication capabilities, music storage and playback devices with wireless communication capabilities, internet devices enabling wireless internet access and browsing, tablets with wireless communication capabilities, and portable units or terminals incorporating combinations of such capabilities.

[0117] Furthermore, various embodiments of any of these devices can be used in UE vehicles associated with a ground network, high-altitude platform stations, or any other such type of node, or in any drone-type radio, aircraft or other aircraft-mounted radio, or on waterborne vehicles such as ships.

[0118] An exemplary embodiment of the present invention provides at least an improved ALF pipeline that uses CCCM to locally map the improved lumens of the ALF to the improved chromens. The output of the CCCM filtering stage is then used as input to both the ALF chromens and the CC-ALF. The exemplary embodiment of the present invention also provides several embodiments to address more specific cases.

[0119] In VVC, the ALF uses a Wiener filter to reduce the MSE of the lumar and chroma reconstructions. The filter coefficients are selected from a fixed set of coefficients using signaled indices, or they are explicitly signaled from the encoder to the decoder. In both cases, signaling is involved. A variation of the cross-component called CC-ALF improves the chroma by applying a Wiener filter to the lumar component to obtain a correction term for the chroma component (see Figure 10). In all ALF-related filter derivations, the goal (on the encoder side) is to minimize the error relative to the original lumar and chroma.

[0120] The VCC ALF pipeline in the decoder can be summarized in the following steps: 1a. Apply Ruma ALF to the reconstructed Ruma. 2a. Apply Chroma ALF to the reconstructed chroma. 3a. To obtain chroma correction, apply CC-ALF to the reconstructed chroma (i.e., the input from step 1a). 4a. Add the output of step 3a to the output of step 2a.

[0121] In an exemplary embodiment of the present invention, an improved ALF pipeline is introduced, as shown in Figure 11.

[0122] Figure 11 shows the improved ALF pipeline using CCCM-based chroma update.

[0123] In the illustrated embodiment, the luma input to CC-ALF is the output of the ALF luma stage, but it can also be configured to be the output of the SAO luma stage. Furthermore, depending on the embodiment, the derivation of the CCCM model can also be placed before the SAO stage. Any input path to the CCCM filtering stage is used when some of the output samples of the CCCM filtering stage are blended with the SAO output samples, or when some of the CCCM filtering output samples are made exactly the same as the SAO output samples. It should be understood that all illustrated CCCM stages can be replaced with any cross-component prediction tool such as CCLM, GL-CCCM, or any other cross-component prediction tool, as needed.

[0124] Most importantly, an additional filtering stage is inserted between the ALF lumens stage and the ALF chromatens stage.

[0125] Because this filtering stage is based on the CCCM method, it does not require signaling and uses a different minimization target than ALF.

[0126] In CCCM, the minimization goal is the reconstructed chrominance, not the original chrominance, and the model derivation is also performed on the decoder side. The following steps describe the improved pipeline on the decoder side. 1b. Obtain a CCCM filter with high spatial granularity using the reconstructed lumens and chromens as reference samples. 2b. Apply Ruma ALF to the reconstructed Ruma. 3b. Using the output of step 2b as input, apply the CCCM filter from step 1b. 4b. Apply Chroma ALF to the output of step 3b. 5b. To obtain chroma correction, apply CC-ALF to either a) the reconstructed chroma, or b) the output of step 2b. 6b. Add the output of step 5b to the output of step 4b.

[0127] The output of step 2b is a higher quality chroma than the one used as input in step 1b. If a higher quality chroma is used as input in step 3b, essentially a higher quality chroma version can be obtained as output according to exemplary embodiments of the present invention. CCCM filtering preserves the color space characteristics (obtained by the model in step 1b) but converts luminance corrections (such as edge / gradient correction or texture smoothing / sharpening) into a higher quality chroma version.

[0128] One of the main advantages of the improved ALF pipeline is the granularity at which the CCCM model is derived and filtered. The CCCM model is derived and applied to small blocks that can be independent of the coding and prediction partitions. For example, the blocks may be 1x1, 2x2, 4x4, etc., and can have a square or rectangular shape. The blocks may be overlapping or distinct. Compared to CC-ALF filters at the CTU level, the proposed method can track the cross-component model with high spatial accuracy, thus mapping detailed lumen improvements to chroma improvements in finer detail. This is in contrast to CC-ALF, where filter coefficients are only changed at the CTU level and only eight types of filters can be used per picture. With CCCM, thousands of types of filters can be used, for example, one for every 8x8 blocks, without additional signaling costs.

[0129] In the proposed improved pipeline, CC-ALF can still be applied, and since the CC-ALF filter is derived in the encoder by minimizing the squared error relative to the original sample, it still offers advantages.

[0130] In step 5b above, depending on the embodiment, the input to the CC-ALF may be either an unfiltered reconstructed rumor (i.e., the input in step 1b) or an ALF-filtered rumor (i.e., the output in step 2b).

[0131] In the filtering stage in step 3b, the performance of the CCCM model can be considered. If the model's MSE is obtained during model derivation and the MSE is considered high, the CCCM stage for a given block can be skipped.

[0132] It should be understood that all of the aforementioned CCCM stages can be replaced, as needed, with any cross-component prediction tool such as CCLM, GL-CCCM, or any other cross-component prediction tool. For clarity, the following embodiments use the state-of-the-art cross-component prediction tool CCCM as an example.

[0133] In one embodiment, the derivation and filtering of the CCCM model can be performed in blocks of any given granularity, such as 1x1, 2x2, 4x4, 8x8, 16x16, or using rectangular blocks such as 4x8 or 8x4.

[0134] In one embodiment, the blocks to which the CCCM model is derived and applied may be separate or overlapping.

[0135] In one embodiment, CCCM blocks may depend on or be independent of the encoding or prediction partitioning. For example, instead of using very fine granularity, the blocks may follow a chroma or luma partitioning with some degree of precision, thereby speeding up model derivation and filtering.

[0136] In one embodiment, during the derivation of a CCCM model using MSE, the model can be checked block by block, and if the MSE exceeds a given threshold, the CCCM stage for that block can be skipped.

[0137] In embodiments based on the previous model, the MSE threshold may be fixed or signaled from the encoder to the decoder.

[0138] In embodiments based on the previous embodiment, the MSE threshold can be estimated based on the underlying coded partitioning, lumens sample values, or chromens sample values.

[0139] In one embodiment, the CCCM model can have any number of filter coefficients.

[0140] In one embodiment, the derivation and filtering of the CCCM model can be replaced with a simplified variation such as CCLM.

[0141] In one embodiment, the derivation and filtering of the CCCM model can be replaced with more advanced variations that take into account gradient and positional information (such as GL-CCCM).

[0142] In one embodiment, the CCCM model derivation stage can be placed before or after the SAO stage.

[0143] In one embodiment, the CCCM model derivation stage can be placed at any point before the Luma ALF stage.

[0144] In one embodiment, the luma input to the CC-ALF stage can be either an input to the ALF luma stage or an output to the ALF luma stage.

[0145] In one embodiment, the filter may consist of auxiliary information to guide the filter to further improve a region or sample having specific characteristics. Examples of such auxiliary information are as follows: ●The input to the filter may include residual information from the rumor block. ●The input to the filter may include prediction information for the rumor block. ● The input to the filter may include one or more of the transformation coefficients of the Ruma Block. ● The input to the filter may include the input and / or output of a previous filtering operation. For example, the input and / or output of a deblocking filter, the input and / or output of an SAO filter, the input and / or output of an ALF filter, etc. ● The input to the filter may include the difference between the input and output of a previous filtering operation. For example, the difference between the input and output of a deblocking filter, the difference between the input and output of an SAO filter, the difference between the input and output of an ALF filter, etc. Furthermore, a scaling operation may be applied to the difference values ​​before they are input to the CCCM filter.

[0146] In one embodiment, the use of the CCCM stage can be signaled from the encoder to the decoder.

[0147] In one embodiment, the use of filters can be signaled at different granularities; for example, signaling may be performed for each filtering unit, such as 1x1, 2x2, etc. Alternatively, signaling may be performed at a different granularity than that of filtering. For example, filter activation may be performed at a coarser granularity for each CTU, and if a filtering mode is enabled for that CTU, filtering may be performed at a finer granularity (e.g., 1x1, 2x2, 4x4, etc.). In this case, the encoder may decide whether to use filters for the entire CTU based on rate distortion optimization.

[0148] In one embodiment, different types and / or sizes of filters such as conventional CCCM, GL-CCCM, CCLM, or any other variations may be determined and signaled on a per-CTU, slice, sub-picture, picture, or sequence level basis.

[0149] In one embodiment, the CCCM stage can be applied to all chromatic components or a specified set of chromatic components.

[0150] In one embodiment, the proposed ALF pipeline can be used in any color space.

[0151] In one embodiment, an additional chroma-versus-chroma CCCM stage can be inserted between the ALF Cb stage and the ALF Cr stage, thereby making the chroma ALF sequential as well.

[0152] In one embodiment, the CCCM model derivation and filtering stages can use downsampled lumens and / or the original lumens samples.

[0153] In one embodiment, the filtering units may consist of non-overlapping blocks. In this case, the filter derivation process for all filtering units may be performed in parallel. Furthermore, once the parameters for a particular unit are derived, CCCM filtering can be started in the pipeline without significant delay.

[0154] Figure 13 illustrates, but is not limited to, operations that may be performed by a device, such as UE10 in Figure 12. As shown in step 1310 of Figure 13, at least one convolutional cross-component model filter may be obtained for the sample set. As shown in step 1320 of Figure 13, the sample set is reconstructed samples of two channels of the image. As shown in step 1330 of Figure 13, at least one filter may be applied to the reconstructed sample set of the first channel. As shown in step 1340 of Figure 13, at least one filter may be applied to the reconstructed sample set of the first channel. As shown in step 1350 of Figure 13, at least one convolutional cross-component model filter may be applied to the reconstructed sample set of the first channel using the output of at least one filter as input. As shown in step 1360 of Figure 13, at least one filter may be applied to the output of at least one convolutional cross-component model filter. As shown in step 1370 of Figure 13, the output of at least one convolutional cross-component model filter may be used as input to apply a cross-component filter to the reconstructed set of samples of the first channel, or the cross-component filter may be applied to the output of at least one filter to obtain a correction to the output of at least one filter.

[0155] According to the exemplary embodiments described in the paragraph above, the output of at least one convolutional cross-component model filter may be used as an input to add the output of the cross-component filter to the output of at least one filter.

[0156] According to the exemplary embodiment described in the paragraph above, a convolutional cross-component model filter is derived and applied to blocks that are not determined by coding or predicting partitions.

[0157] According to the exemplary embodiments described in the paragraph above, the blocks may have a square or rectangular shape, and the blocks may overlap or be separate. The use of a convolutional cross-component model for each block is determined without additional signaling.

[0158] A non-temporary computer-readable medium (MEM12B shown in Figure 12) stores program code (PROG10C shown in Figure 12), which is executed by at least one processor (DP10A and / or DP10F shown in Figure 12) to perform at least the operations described in the above paragraph.

[0159] According to the exemplary embodiment of the present invention described above, there is an apparatus (one or more transceivers 10D and / or one or more transceivers 13D, MEM10B and / or MEM13B, PROG10C and / or PROG13C, and DP10A and / or DP13A in Figure 12) that takes the output of at least one convolutional cross-component model filter as an input and adds the output of the cross-component filter to the output of at least one filter, wherein the convolutional cross-component model filter is derived and applied to a block that is not determined by any coding or predictive partition (one or more transceivers 10D in Figure 12) (and / or one or more transceivers 13D, MEM10B and / or MEM13B, PROG10C and / or PROG13C, and DP10A and / or DP13A), the blocks have a square or rectangular shape, the blocks may overlap or be separate, and the use of a convolutional cross-component model for each block is determined without additional signaling (one or more transceivers 10D and / or one or more transceivers 13D, MEM10B and / or MEM13B, PROG10C and / or PROG13C, and DP10A and / or DP13A in Figure 12).

[0160] In exemplary embodiments of the present invention as described in the paragraphs above, the means for adding, deriving, applying, and determining comprises at least a non-temporary computer-readable medium (MEM10B and / or MEM13B in Figure 12) encoded using a computer program (PROG10C and / or PRPG13C in Figure 12) executable by at least one processor (DP10A and / or DP13A in Figure 12).

[0161] Furthermore, according to exemplary embodiments of the present invention, a circuit is provided for performing the operation according to the exemplary embodiments of the present invention disclosed herein. This circuit may include any type of circuit, such as a content encoding circuit, a content decoding circuit, a processing circuit, an image generation circuit, or a data analysis circuit. Furthermore, this circuit may include discrete circuits, application-specific integrated circuits (ASICs), and / or field-programmable gate array circuits (FPGAs), as well as a dual-core processor with a processor specifically configured to perform its respective functions by software, or a digital signal processor corresponding to the software. Furthermore, necessary inputs to and outputs from the circuit, the functions performed by the circuit, and interconnections (possibly via inputs and outputs) between the circuit and other components, which may include other circuits, are provided for performing the exemplary embodiments of the present invention described herein.

[0162] According to exemplary embodiments of the present invention disclosed in this application, the “circuit” provided may include at least one or more, or all of, the following: (a) Implementation of circuits using only hardware (such as implementation using only analog and / or digital circuits), (b) A combination of hardware circuits and software, for example (if applicable): (i) combinations of analog and / or digital hardware circuits and software / firmware, and (ii) Any part of a hardware processor, software, and memory having software (including a digital signal processor) that works in conjunction with a device such as a mobile phone or server to perform various functions such as functions or operations according to exemplary embodiments of the present invention disclosed herein, and (c) Hardware circuits and / or processors, such as a microprocessor or part of a microprocessor, that require software (e.g., firmware) to operate, but may not be present if such software is not necessary for operation.

[0163] According to exemplary embodiments of the present invention, there exists sufficient circuitry to perform at least the novel operations described in the embodiments of the present invention disclosed herein, and as used herein, “circuitry” means at least the following: (a) Hardware-only circuit implementation (such as implementation in analog and / or digital circuits only), and (b) combinations of circuitry and software (and / or firmware), for example (where applicable): (i) combinations of processors, or (ii) a portion of processors / software (including digital signal processors), software, and memory that work together to cause a device such as a mobile phone or server to perform various functions, (c) A circuit such as a microprocessor or part of a microprocessor that requires software or firmware to operate, even if the software or firmware is not physically present.

[0164] This definition of “circuit” applies to all uses of the term in this application, including in the claims. As a further example, as used in this application, the term “circuit” also encompasses a mere processor (or more processors) or a part of a processor and / or its associated software and / or firmware implementation. The term “circuit” also, as applied to a particular claim element, encompasses, for example, a baseband integrated circuit or application processor integrated circuit for a mobile phone, or a similar integrated circuit in a server, cellular network device, or other network device.

[0165] Generally, various embodiments can be implemented by hardware or dedicated circuitry, software, logic, or any combination thereof. For example, some embodiments may be implemented in hardware, while others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, but the present invention is not limited to these. Various embodiments of the present invention may be described and illustrated using block diagrams, flowcharts, or any other illustrative representation, but it is well understood that these blocks, apparatus, systems, techniques, or methods described herein can be implemented, in non-limiting examples, by hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers, or other computing devices, or a combination thereof.

[0166] Embodiments of the present invention can be implemented in various components, such as integrated circuit modules. Designing integrated circuits is generally a highly automated process. Complex and powerful software tools are available to translate logic-level designs into semiconductor circuit designs that can be etched and formed on semiconductor substrates.

[0167] As used herein, the term "exemplary" means "serving as an example, instance, or illustration". Embodiments described as "exemplary" in this specification should not necessarily be construed as being more preferred or advantageous than other embodiments. All of the embodiments described in the mode for carrying out the invention are exemplary embodiments provided to enable those skilled in the art to implement or use the invention, and do not limit the scope of the invention defined by the claims.

[0168] The above description has provided a complete and helpful illustration, by way of non-limiting and exemplary examples, of the best methods and apparatuses currently contemplated by the inventors for carrying out the invention. However, it will become apparent to those skilled in the relevant art that various modifications and adaptations are possible when the above description is read in conjunction with the accompanying drawings and the appended claims. However, such changes and similar modifications to the teachings of the exemplary embodiments of the invention are still within the scope of the invention.

[0169] Note that the terms "connected", "coupled", or variations thereof mean any direct or indirect connection or coupling between two or more elements, and may also include the presence of one or more intermediate elements between the two "connected" or "coupled" elements. The coupling or connection between elements may be physical, logical, or a combination thereof. In this specification, as some non-limiting and comprehensive examples, two elements can be considered "connected" or "coupled" by connection with one or more conductors, cables, and / or printed wirings, and also by the use of electromagnetic energy having wavelengths in, for example, radio frequency regions, microwave regions, and optical (including visible and invisible light) regions.

[0170] Furthermore, some features of preferred embodiments of the present invention can be advantageously used without corresponding to other features. Therefore, the above description is merely illustrative of the principles of the present invention and should not be considered as a limitation thereof.

Claims

1. At least one processor, When executed by the aforementioned at least one processor, the device will have This involves obtaining at least one convolutional cross-component model filter for a set of samples, The aforementioned set of samples is obtained as reconstructed samples of two channels of the image, Applying at least one filter to the set of reconstructed samples of the first channel, Applying the at least one convolutional cross-component model filter to the set of reconstructed samples of the first channel using the output of the at least one filter as an input, Applying at least one filter to the output of the at least one convolutional cross-component model filter, To obtain a correction to the output of the at least one convolutional cross-component model filter, the cross-component filter is applied to the set of reconstructed samples of the first channel, or the cross-component filter is applied to the output of the at least one filter to the set of reconstructed samples of the first channel. At least one non-temporary memory that stores instructions that cause at least one to execute and A device equipped with the following features.

2. The at least one non-temporary memory that stores instructions is executed by the at least one processor, and the device, The apparatus according to claim 1, wherein the output of the at least one convolutional cross-component model filter is used as an input to add the output of the cross-component filter to the output of the at least one filter.

3. The apparatus according to claim 1, wherein the convolutional cross-component model filter is derived and applied to blocks not determined by coding or predictive partitioning.

4. The apparatus according to claim 3, wherein the blocks have a square or rectangular shape, and the blocks may overlap or be separate.

5. The apparatus according to claim 3, wherein the use of the convolutional cross-component model for each block is determined without additional signaling.

6. This involves obtaining at least one convolutional cross-component model filter for a set of samples, The aforementioned set of samples is obtained as reconstructed samples of two channels of the image, Applying at least one filter to the set of reconstructed samples of the first channel, Applying the at least one convolutional cross-component model filter to the set of reconstructed samples of the first channel using the output of the at least one filter as an input, Applying at least one filter to the output of the aforementioned at least one convolutional cross-component model filter, To obtain a correction to the output of the at least one convolutional cross-component model filter, the cross-component filter is applied to the set of reconstructed samples of the first channel, or the cross-component filter is applied to the output of the at least one filter to the set of reconstructed samples of the first channel. A method that includes [a certain feature].

7. The at least one non-temporary memory that stores instructions is executed by the at least one processor, and the device, The apparatus according to claim 6, wherein the output of the at least one convolutional cross-component model filter is used as an input to add the output of the convolutional cross-component filter to the output of the at least one filter.

8. The method according to claim 6, wherein the convolutional cross-component model filter is derived and applied to blocks that are not determined by coding or predicting partitions.

9. The method according to claim 8, wherein the blocks have a square or rectangular shape, and the blocks may be overlapping or separate.

10. The method according to claim 8, wherein the use of the convolutional cross-component model for each block is determined without additional signaling.

Images