High-granularity decoder-side cross-component loop filter

Abstract

In accordance with an example embodiment of the present invention there is at least one method and apparatus to perform: obtaining at least one convolutional cross-component model filter for a sample set, where the sample set is reconstructed samples for two channels of an image; applying at least one filter to the reconstructed sample set of the first channel; applying at least one convolutional cross-component model filter using an output of at least one filter for a reconstructed sample set of the first channel as an input; applying at least one filter to an output of the at least one convolutional cross-component model filter; and applying a cross-component filter to the reconstructed sample set of the first channel, or to the output of at least one filter for the reconstructed sample set of the first channel, to obtain a correction for the output of the at least one filter using the output of the at least one convolutional cross-component model filter as input, as visible in FIG. 13.

Description

Technical Field

[0001] The teachings of exemplary embodiments of the present invention generally relate to video encoding and decoding, and more specifically to improved chroma construction. Background Technology

[0002] This section is intended to provide background or context for the invention referenced in the claims. The description herein may include concepts that may be pursued, but not necessarily concepts that have been previously conceived or pursued. Therefore, unless otherwise stated herein, what is described in this section is not prior art to the specification and claims of this application, and is not acknowledged as prior art by virtue of its inclusion in this section.

[0003] Some abbreviations that may be found in the instruction manual and / or accompanying drawings are defined here as follows: ALF Adaptive Loop Filter AMVR Adaptive Motion Vector Resolution CC cross component CC-ALF Cross Component Adaptive Loop Filter CCLM Cross Component Linear Model Intra-Frame Prediction CTU coding tree unit CU Central Unit ISM Industrial Science and Medicine ISP intra-frame sub-partition LM linear model LMS (Least Mean Square) MRL Multiple Reference Lines MMLM (Multi-Model LM) MVD Motion Vector Difference VVC Multi-Functional Video Codec WP Weighted Prediction

[0004] Brief description of previous developments

[0005] Block-based processing is widely used in video coding because it offers a good trade-off between coding efficiency and computational complexity. Intra-frame block copying tools are known to generate predictions for the current block.

[0006] The exemplary embodiments of the present invention propose improved operations for model monitoring processes, such as those used for beam prediction. Summary of the Invention

[0007] This section contains examples of possible implementations, but is not intended to limit you.

[0008] In another exemplary aspect of the invention, there is an apparatus, such as a user equipment side apparatus, comprising: at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: acquire at least one convolutional cross-component model filter for a sample set, wherein the sample set is reconstructed samples for two channels of an image; apply the at least one filter to the reconstructed sample set of a first channel; apply the at least one convolutional cross-component model filter using the output of the at least one filter for the reconstructed sample set of the first channel as input; apply the at least one filter to the output of the at least one convolutional cross-component model filter; and apply the cross-component filter to the reconstructed sample set of the first channel, or to the output of the at least one filter for the reconstructed sample set of the first channel, to acquire a correction for the output of the at least one filter using the output of the at least one convolutional cross-component model filter as input.

[0009] In another exemplary aspect of the invention, there is a method comprising: obtaining at least one convolutional cross-component model filter for a sample set, wherein the sample set is reconstructed samples for two channels of an image; applying at least one filter to the reconstructed sample set for a first channel; applying at least one convolutional cross-component model filter as input using the output of the at least one filter in the reconstructed sample set for the first channel; applying at least one filter to the output of the at least one convolutional cross-component model filter; and applying a cross-component filter to the reconstructed sample set for the first channel, or to the output of the at least one filter for the reconstructed sample set for the first channel, to obtain a correction for the output of the at least one filter using the output of the at least one convolutional cross-component model filter as input.

[0010] Another example embodiment is an apparatus and method that includes the apparatus and method described in the preceding paragraph, wherein the output of at least one convolutional cross-component model filter is used as input to add the output of the cross-component filter to the output of at least one filter, wherein the convolutional cross-component model filter is derived and applied to blocks not determined by any coding or prediction partitions, wherein the blocks have a square or rectangular shape, and wherein the blocks may be overlapping or distinct, wherein the use of the convolutional cross-component model for each block is determined without additional signaling.

[0011] A non-transitory computer-readable medium storing program code that is executed by at least one processor to perform at least the methods described in the preceding paragraphs.

[0012] In another exemplary aspect of the invention, there is an apparatus comprising: a component for adding the output of a cross-component filter to the output of at least one filter using the output of at least one convolutional cross-component model filter as input, wherein the convolutional cross-component model filter is derived and applied to blocks not determined by any coding or prediction partitions, wherein the blocks have a square or rectangular shape, and wherein the blocks may be overlapping or distinct, wherein the use of a convolutional cross-component model for each block is determined without additional signaling.

[0013] According to the example embodiments described in the preceding paragraphs, at least the components used for adding, exporting, applying, and determining include: a network interface, and computer program code stored on a computer-readable medium and executable by at least one processor.

[0014] The communication system includes network-side devices and user equipment-side devices that perform the above operations. Attached Figure Description

[0015] The above and other aspects, features, and benefits of the various embodiments of this disclosure will become more fully apparent from the following detailed description with reference to the accompanying drawings, wherein like reference numerals are used to denote like or equivalent elements. The drawings are shown to facilitate a better understanding of the embodiments of this disclosure and are not necessarily drawn to scale. In the drawings:

[0016] Figure 1A The locations of the samples used to derive α and β are shown;

[0017] Figure 1B Table 1 shows the chromaticity prediction mode derived from the luminance mode when cclm_ is enabled;

[0018] Figure 1C Table 2 shows a unified binary table for chromaticity prediction modes;

[0019] Figure 2 This illustrates the classification of luminance samples into two spatial domains (top and bottom of the sample domain) for deriving two sets of α and β.

[0020] Figure 3 This shows the locations of the samples used to derive the CCCM filter when six reference lines are used; and

[0021] Figure 4 From the left side: three taps vertical, three taps horizontal, five taps intersecting, twenty-five taps diamond;

[0022] Figure 5 An example of four reference lines adjacent to the prediction block is shown;

[0023] Figure 6 The matrix-weighted intra-frame prediction process is illustrated.

[0024] Figure 7 The top and left adjacent blocks used in the CIIP weight derivation are shown;

[0025] Figure 8 The ALF filter shape is shown (chroma: 5x5 rhombus, luminance: 7x7 rhombus);

[0026] Figure 9A The calculation of the quadratic sampling Laplace operator is shown;

[0027] Figure 9B A table showing the mappings of gradients and transformations computed for a block is provided;

[0028] Figure 10 The placement of (a) CC-ALF relative to other loop filters and (b) diamond-shaped filter are shown.

[0029] Figure 11 And an improved ALF pipeline with CCCM-based chromaticity updates;

[0030] Figure 12 A block diagram of a possible, and non-limiting, exemplary system in which this example embodiment may be practiced is shown; and

[0031] Figure 13 A method that can be performed by an apparatus according to an example embodiment of the present invention is shown. Detailed Implementation

[0032] In exemplary embodiments of the present invention, at least one method and apparatus are proposed for at least improving the chroma construction for video encoding and decoding.

[0033] Hybrid video codecs (e.g., ITU-T H.263, H.264 / AVC, and HEVC) can encode video information in two stages. First, pixel values ​​in a picture (or "block") are predicted, for example, by a motion compensation component (which finds and indicates the region closely corresponding to the block being encoded in a previously encoded video frame) or by a spatial component (which uses the pixel values ​​surrounding the block to be encoded in a specified manner). In this first stage, predictive coding can be applied, for example, to so-called sample prediction and / or so-called syntax prediction.

[0034] In sample prediction, pixel or sample values ​​within a region or "block" of an image are predicted. For example, these pixel or sample values ​​can be predicted using one or more of motion compensation or intra-frame prediction mechanisms.

[0035] Motion compensation mechanisms (also known as inter-frame prediction, temporal prediction, or motion-compensated temporal prediction, or motion-compensated prediction, or MCP) involve finding and indicating regions in a previously encoded video frame that closely correspond to the block being encoded. Inter-frame prediction can reduce practical redundancy.

[0036] Intra-frame prediction, where pixel or sample values ​​are predicted using spatial mechanisms, involves finding and indicating spatial region relationships. Intra-frame prediction leverages the fact that neighboring pixels within the same image may be related. Intra-frame prediction can be performed in the spatial or transform domain; that is, sample values ​​or transform coefficients can be predicted. Intra-frame prediction is typically used in intra-frame coding, where no inter-frame prediction is applied.

[0037] In syntax prediction, also known as parameter prediction, syntax elements and / or syntax element values ​​and / or variables derived from syntax elements are predicted from earlier (de-encoded) syntax elements and / or earlier derived variables. Non-restrictive examples of syntax prediction are provided below.

[0038] In motion vector prediction, for example, motion vectors used for inter-frame and / or inter-view prediction can be differentially encoded relative to the predicted motion vectors of a specific block. In many video codecs, predicted motion vectors are created in a predetermined manner, for example, by calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another method for creating motion vector predictions (sometimes called Advanced Motion Vector Prediction (AMVP)) is to generate a list of candidate predictions from adjacent blocks and / or blocks at the same location in a time-referenced picture, and to signal the selected candidate as the motion vector predictor. In addition to predicting motion vector values, reference indices from previously encoded / decoded pictures can be predicted. Reference indices are typically predicted from adjacent blocks and / or blocks at the same location in a time-referenced picture. Differential encoding of motion vectors is typically disabled across slice boundaries.

[0039] For example, block partitions from CTU to CU, and down to PU, can be predicted.

[0040] In filter parameter prediction, for example, filter parameters used for sample adaptive offset can be predicted.

[0041] Prediction methods that use image information from previously encoded images can also be called inter-frame prediction methods, which can also be referred to as temporal prediction and motion compensation. Prediction methods that use image information within the same image can also be called intra-frame prediction methods.

[0042] Secondly, the prediction error, i.e., the error between the predicted pixel block and the original pixel block, is encoded. This can be done by transforming the difference in pixel values, the quantization coefficients, and the coefficients quantized by entropy coding using a specified transform (e.g., Discrete Pre-Transform (DCT) or a variant thereof). By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size at the transmission bit rate).

[0043] In many video codecs, including H.264 / AVC and HEVC, motion information is indicated by motion vectors associated with each motion-compensated image block. Each of these motion vectors represents the image block to be encoded (in the encoder) or decoded (at the decoder) in the picture, as well as the displacement of the predicted source block in one of the previously encoded or decoded pictures (or images). In H.264 / AVC and HEVC, as with many other video compression standards, the picture is divided into a rectangular grid, and for each rectangular grid, a similar block in a reference picture is indicated for inter-frame prediction. The position of the predicted block is encoded as a motion vector that indicates the position of the predicted block relative to the block being encoded.

[0044] The following new encoding tools exist in the Multi-Functional Video Codec (VCC) currently under development (more details will be added in the final draft patent if needed): • Intra-frame prediction: – 67 intra-frame modes with wide-angle mode extension; – Block size and mode-dependent four-tap interpolation filter; – Location-dependent intra-prediction combination (PDPC); – Cross-component linear model intra-frame prediction (CCLM). – Intra-frame prediction with multiple reference lines; – Intra-frame sub-partition; - Weighted intra-frame prediction with matrix multiplication; • Image prediction: – Block motion replicas with spatial, temporal, historical, and pairwise average merging candidates; – Affine motion inter-frame prediction; - Sub-block-based temporal motion vector prediction; - Adaptive motion vector resolution; – 8x8 block-based motion compensation for temporal motion prediction; – High-precision (1 / 16 pixel) motion vector storage and motion compensation with an eight-tap interpolation filter for the luminance component and a four-tap interpolation filter for the chrominance component. – Triangular partitioning; – Combine intra-frame and inter-frame prediction; – Merge with MVD (MMVD) – Symmetric MVD encoding; – Bidirectional optical flow; – Decoder-side motion vector optimization; – Two-way prediction with CU level weights; • Transformation, quantization, and coefficient encoding: – Offers multiple master transform options: DCT2, DST7, and DCT8; – Used for auxiliary conversion in the low-frequency band; – Sub-block transformation used for inter-frame prediction residuals; – Relevant quantization with a maximum QP increasing from 51 to 63; – Transform coefficient encoding with hidden identifier data; – Skip the transformation of residual encoding; • Entropy coding: – An arithmetic coding engine with adaptive dual-window probability updates; In a loop filter: – Loop reshaping; – A deblocking filter with strong, long filters; – Sample adaptive offset; – Adaptive loop filter; • Screen content encoding: – Current image reference with reference area restrictions; • 360-degree video encoding: – Horizontal orbital motion compensation; • Advanced syntax and parallel processing: – Reference image management with direct reference image list signaling; – A tile set with rectangular tile groups;

[0045] Partitioning in VVC

[0046] In VVC, each image is divided into coding tree units (CTUs) similar to HEVC. Images can also be divided into slices, tiles, blocks, and sub-images. CTUs can be further subdivided into smaller CUs using a quadtree structure. Each CU can be partitioned using quadtrees and nested multi-type trees that include ternary and binary partitions.

[0047] There are specific rules for inferring partitions within image boundaries.

[0048] Redundant partitioning patterns are not allowed in nested multi-type partitions.

[0049] Cross-component linear model prediction (CCLM)

[0050] To reduce cross-component redundancy, a cross-component linear model (CCLM) prediction mode is used in VCC. For this prediction mode, chroma samples are predicted based on reconstructed luminance samples from the same CU using the following linear model: in, This represents the predicted chromaticity sample in the CU, and This represents the reconstructed brightness sample taken down from the same CU.

[0051] The CCLM parameters (α and β) are derived from up to four adjacent chroma samples and their corresponding downsampled luminance samples. Assuming the current chroma block size is W×H, then W' and H' are set as follows: - When the LM pattern is applied, W'=W, H'=H; - When the LM-A pattern is applied, W' = W + H; - When the LM-L pattern is applied, H' = H + W;

[0052] The adjacent positions mentioned above are denoted as S[0, -1]…S[W'-1, -1], and the adjacent positions to the left are denoted as S[-1, 0]…S[-1, H'-1]. Then the four samples are selected as follows: - When the LM mode is applied, and both the upper and left adjacent samples are available, - When LM-A mode is applied, or only the upper adjacent sample is available, - When the LM-L mode is applied, or only the left adjacent sample is available,

[0053] Four adjacent luminance samples at the selected location are downsampled and compared four times to find two smaller values: x0A and x1A, and two larger values: x0B and x1B. Their corresponding chromaticity sample values ​​are represented as y0A, y1A, y0B, and y1B. Then xA, xB, yA, and yB are derived as follows:

[0054] Finally, the linear model parameters α and β are obtained according to the following equation:

[0055] Figure 1 shows an example of the positions of the left and top samples, as well as the sample in the current block, involved in CCLM mode.

[0056] The division operation for calculating the parameter α is implemented using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are represented using exponent notation. For example, diff is approximated using a 4-bit significant part and an exponent. Therefore, the table for 1 / diff is simplified to 16 elements for the 16 values ​​of the significant bits, as follows:

[0057] This will have the benefit of reducing computational complexity and the amount of memory required to store the necessary tables.

[0058] In addition to the top and left templates being used together to calculate linear model coefficients, they can also be used alternatively in two other LM modes, referred to as LM_A and LM_L modes.

[0059] In LM_A mode, only the top template is used to calculate the linear model coefficients. To obtain more samples, the top template is extended to (W+H). In LM_L mode, only the left template is used to calculate the linear model coefficients. To obtain more samples, the left template is extended to (H+W).

[0060] For non-square blocks, the top template is expanded to W+W, and the left template is expanded to H+H.

[0061] To match the chroma sample positions for a 4:2:0 video sequence, two types of downsampling filters are applied to the luminance samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions. The selection of the downsampling filter is specified by the SPS level flag. The two downsampling filters are shown below, corresponding to "Type 0" and "Type 2" content, respectively.

[0062] Note that when the upper reference line is at the CTU boundary, only one luminance line (the universal line buffer in intra-frame prediction) is used for downsampling luminance samples.

[0063] This parameter calculation is performed as part of the decoding process, and not just as part of the encoder search operation. As a result, no syntax is used to pass the α and β values ​​to the decoder.

[0064] For chroma intra-mode coding, a total of eight intra-mode modes are allowed. These modes include five traditional intra-mode modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). Figure 1B Table 1 illustrates the chroma mode signaling and derivation process. Chroma mode encoding directly depends on the intra-prediction mode of the corresponding luma block. Because separate block partitioning structures for luma and chroma components are enabled in the I-slice, one chroma block can correspond to multiple luma blocks. Therefore, for chroma DM mode, the intra-prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited.

[0065] exist Figure 1C In Table 2, the first bit indicates whether it is normal mode (0) or LM mode (1). If it is LM mode, the next bit indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next 1 bit indicates whether it is LM_L (0) or LM_A (1). In this case, when sps_cclm_enabled_flag is 0, the first bin of the binary table corresponding to intra_chroma_pred_mode can be discarded before entropy encoding. Or, in other words, the first bit is inferred to be 0 and therefore not encoded. This single binary table is used for the case where sps_cclm_enabled_flag is equal to both 0 and 1. The first two bits in (Error! Reference not found) are context-encoded with their own context model, and the remaining bits are bypassed.

[0066] Furthermore, to reduce luma-chroma latency in the dual tree, when the 64x64 luma coding tree node is partitioned by no splitting (and ISP is not used for 64x64 CU) or QT, the chroma CU in the 32x32 / 32x16 chroma coding tree node is allowed to use CCLM in the following manner: - If the 32x32 chroma node is not split or QT partitioned, then all chroma CUs in the 32x32 node can use CCLM; - If a 32x32 chroma node is partitioned by a horizontal BT, and the 32x16 child node is not partitioned or is partitioned by a horizontal BT, then all chroma CUs in the 32x16 chroma node can use CCLM.

[0067] CCLM is not allowed for chroma CU in all other luma and chroma coding tree segmentation conditions.

[0068] Multi-model LM (MMLM)

[0069] The CCLM included in VVC is extended by adding three multi-model LM (MMLM) modes. In each MMLM mode, reconstructed neighbor samples are classified into two classes using a threshold that is the average of the brightness reconstructed neighbor samples. The linear model for each class is derived using the least mean square (LMS) method. For the CCLM mode, the LMS method is also used to derive the linear model. Figure 2 Two luminance-to-chrominance models are shown for obtaining a luminance (Y) threshold of 17. Each luminance-to-chrominance model has its own linear model parameters α and β. As can be seen from the lower part of the figure, each luminance-to-chrominance model corresponds to a spatial segmentation of the content (i.e., its corresponding different objects or textures in the scene).

[0070] Convolutional Cross-Component Model (CCCM)

[0071] An improved version of cross-component prediction, known as CCCM, uses a 2D filter kernel to derive a luma-to-chroma model. Filter coefficients are derived on the decoder side using a reconstructed set of input data and chroma samples. For filter coefficient derivation, a reference sample region at the same location (consisting of reconstructed luma and chroma samples) is used for... Figure 3 The brightness and chromaticity are both defined as shown, with the commonly used 4:2:0 chromaticity downsampling being applied. For example, the reference sample area for a given block could be... Figure 3 The six lines at the top and left, however, any number of reference lines (which can be implemented by both the encoder and decoder) can be used. Typically, the reference samples can contain any chroma and luminance samples that have already been reconstructed by both the encoder and decoder. Once the reference samples are determined, the filter coefficients can be derived, for example, using different types of linear regression tools such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or minimum absolute shrinkage and selection operators.

[0072] The size of the filter kernel can be, for example... (1D Vertical) (1D level) , Or any size, and can be shaped (by selecting only a subset of all possible kernel locations) into a cross or a rhombus (such as...). Figure 4 (As shown) or any given shape. When referring to samples within the filter kernel, the following symbols are used: North (top), East (right), South (bottom), West (left), and Center, as shown. Figure 4 As shown, the letters N, E, S, W, and C are used.

[0073] Here, the overall method of reconstructing chroma samples using the convolution between the filter kernels obtained from the decoder side and the input data set is called the Convolutional Cross Component Model (CCCM). The following steps can be applied to perform CCCM operations: 1. Define a reference area at the same location for both the luminance and chrominance components; 2. Perform a step-by-step analysis on the luminance samples to match the chromaticity grid (optional); 3. Scan the luminance and chromaticity samples of the reference region based on the filter shape and collect available statistics (such as autocorrelation matrix and cross-correlation vector). 4. Solve for the filter coefficients by minimizing the squared error (or any other metric) based on available statistics (such as the autocorrelation matrix and cross-correlation vector); 5. The predicted chromaticity patch is calculated by convolving the downsampled luminance samples with the filter kernel.

[0074] Let's define (possibly downsampled) luminance samples as the level to be used. Coordinates and vertical 2D array of coordinate indices Let's also define the chromaticity sample at the same location as a 2D... and the filter kernel (i.e., coefficients) are of At the sample level, the following is defined: The convolution between them is:

[0075] When other data terms are used (such as non-linear square root terms), the attached convolution becomes, in These are the filter coefficients residing outside the 2D filter kernel, which have already been obtained as part of the system of linear equations used to solve for the 2D filter coefficients in step 4 above. Similarly, bias terms can be added to the convolution, through...

[0076] Multi-reference line (MRL) intra-frame prediction

[0077] Multi-reference line (MRL) intra-prediction uses more reference lines for intra-prediction. Figure 5 The example depicts four reference lines, where samples from segments A and F are not extracted from reconstructed neighboring samples, but are instead filled with the nearest samples from segments B and E, respectively. Intra-image prediction in HEVC uses the nearest reference line (i.e., reference line 0). In MRL, two additional lines (reference line 1 and reference line 3) are used.

[0078] The index of the selected reference line (mrl_idx) is signaled and used to generate the intra-predictor. For reference lines idx greater than 0, only additional reference line modes are included in the MPM list, and only the mpm index is signaled if no other modes remain. The reference line index is signaled before the intra-predictor mode, and the planar mode is excluded from the intra-predictor mode if a non-zero reference line index is signaled.

[0079] MRL is disabled for the first line of a block within the CTU to prevent the use of extended reference samples outside the current CTU line. Additionally, PDPC is disabled when additional lines are used. For MRL mode, the derivation of the DC value in the DC intra-prediction mode for non-zero reference line indices is aligned with the derivation of reference line index 0. MRL requires storing three adjacent luma reference lines with the CTU to generate predictions. The Cross Component Linear Model (CCLM) tool also requires three adjacent luma reference lines for its downsampling filters. The definition of MRL using the same three lines is aligned with CCLM to reduce storage requirements for the decoder.

[0080] Intra-frame sub-partition (ISP)

[0081] Depending on the block size, intra-frame sub-partitioning divides the luma intra-frame prediction block vertically or horizontally into 2 or 4 sub-partitions. For example, the minimum block size for ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided into 4 sub-partitions. It should be noted that... (have )and (have ISP blocks can generate blocks with Potential problems with VDPUs. For example, in the case of a single tree, The CU has brightness TB and two corresponding The chromaticity TB. If the CU uses an ISP, the luminance TB will be divided into four. TB (horizontal partitioning is possible only), four Each TB in TB is less than The chroma blocks. However, in the current design of the ISP, chroma blocks are not divided. Therefore, both chroma components will have a greater than [value missing]. The size of the block. Approximately, a similar situation can be handled by using ISP's... The CU is used to create it. Therefore, these two cases are used for The problem lies with the decoder. For this reason, the CU size of the ISP can be limited to a maximum of [missing value]. All sub-partitions satisfy the condition of having at least 16 samples.

[0082] Matrix-weighted intra-frame prediction (MIP)

[0083] Matrix-weighted intra-prediction (MIP) is a new intra-prediction technique added to VVC. It is used to predict the width. and height For the rectangular block samples, matrix-weighted intra-frame prediction (MIP) will be performed on the left row of the block. The row above the reconstructed adjacent boundary samples and blocks The reconstructed neighbor boundary samples are used as input. If the reconstructed samples are unavailable, they are generated in the manner of traditional intra-frame prediction. The generation of the predicted signal is based on the following three steps: averaging, matrix-vector multiplication, and linear interpolation, as follows: Figure 6 As shown.

[0084] Intra-frame prediction in VVC

[0085] The merge list may include the following candidates: 1) Spatial MVP from spatial neighbor CU; 2) MVP of time from the paired CU; 3) History-based MVP from FIFO table; 4) Pairwise average MVP (using candidates already in the list); 5) Zero MV.

[0086] The merge mode width motion vector difference (MMVD) is signaled after the merge candidate is signaled, along with the MVD and resolution index.

[0087] In symmetric MVD, the motion information of list 1 is derived from the motion information of list 0 in the case of double prediction.

[0088] In affine prediction, several motion vectors are indicated / signaled for different angles of the block, and these motion vectors are used to derive the motion vectors of the sub-blocks. In affine merging, the affine motion information of the block is generated based on the normal or affine motion information of the adjacent blocks.

[0089] In sub-block-based temporal motion vector prediction, the motion vectors of the sub-blocks of the current block are predicted from the appropriate sub-blocks in a reference frame, which is indicated by the motion vectors of spatially adjacent blocks (if available).

[0090] In Adaptive Motion Vector Resolution (AMVR), the accuracy of MVD is signaled for each CU.

[0091] In bidirectional forecasting with CU level weighting, the index indicates the weight value used for the weighted average of the two forecast blocks.

[0092] Two-way optical flow (BDOF) refines motion vectors in the case of bidirectional prediction. BDOF generates two prediction blocks using the signaled motion vectors. The gradient values ​​of the two prediction blocks are then used to compute motion refinement to minimize the error between the two blocks. The final prediction block is refined using both the motion refinement and the gradient values.

[0093] Two-way forecasting and weighted forecasting (WP) with CU level weights (BCW).

[0094] In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals acquired from two different reference images and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow for a weighted average of the two prediction signals.

[0095] Five weights are allowed in weighted average two-way forecasting. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) for non-merged CUs, the weight index is signaled after the motion vector difference; 2) for merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is applied only to CUs with 256 or more luminance samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency images, all 5 weights are used. For non-low-latency images, only 3 weights (w∈{3,4,5}) are used. - At the encoder, a fast search algorithm is applied to find the weight indices without significantly increasing encoder complexity. These algorithms are summarized below. For further details, readers can refer to the VTM software and documentation JVET-L0646. When combined with AMVR, if the current image is a low-latency image, unequal weights are conditionally checked only for 1-pixel and 4-pixel motion vector precision. - When combined with an affine, the affine ME will be applied for unequal weights if and only if the affine mode is selected as the current best mode. -When the two reference images in a biprediction are the same, only conditionally check for unequal weights; - Depending on the POC distance between the current image and its reference images, the QP encoding, and the time level, unequal weights will not be searched when a certain condition is met.

[0096] The BCW weight index is encoded using a context-coded bit followed by a bypass-coded bit. The first context-coded bit indicates whether equal weights are used; and if unequal weights are not used, the bypass-coded bit is used to signal the additional bits to indicate which unequal weights were used.

[0097] Weighted Prediction (WP) is an encoding tool supported by the H.264 / AVC and HEVC standards for efficiently encoding video content with fading. Support for WP has also been added to the VVC standard. WP allows signaling of weighting parameters (weights and offsets) for each reference picture in each reference picture list L0 and L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. To avoid interaction between WP and BCW, which would complicate the VVC decoder design, if the CU uses WP, the BCW weight index is not signaled, and w is inferred as 4 (i.e., equal weights are applied). For merging CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to both normal merge mode and inherited affine merge mode. For the constructed affine merge mode, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index for the CU using the constructed affine merge mode is simply set to be equal to the BCW index of the first control point MV.

[0098] In VVC, CIIP and BCW cannot be applied together to a CU. When a CU is encoded in CIIP mode, the BCW index of the current CU is set to 2, for example, with equal weights.

[0099] Combined Inter-Frame and Intra-Frame Prediction (CIIP)

[0100] In VVC, when a CU is encoded in merged mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and CU height are less than 128 luma samples, an additional flag is signaled to indicate whether Combined Inter-Frame / Intra-Frame Prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines inter-frame prediction signals with intra-frame prediction signals. The inter-frame prediction signal in CIIP mode... The same inter-frame prediction process applied to the regular merging mode is used for derivation; and the intra-frame prediction signal is also derived. This is derived after the regular intra-frame prediction process using planar mode. The intra-frame and inter-frame prediction signals are then combined using a weighted average, where the weight values ​​are calculated based on the coding modes of the upper and left adjacent blocks (described in Error! Reference source not found), as follows: - If the adjacent pixel above is available and is intra-coded, set isIntraTop to 1; otherwise, set isIntraTop to 0. - If the left adjacent pixel is available and intra-frame coding is used, set isIntraLeft to 1; otherwise, set isIntraLeft to 0. - If (isIntraLeft+isIntraTop) equals 2, then set wt to 3; Otherwise, if (isIntraLeft+isIntraTop) equals 1, then set wt to 2; Otherwise, set wt to 1.

[0101] Inter-frame prediction (LIC) is a technique that models the local illumination variation between the current block and its predicted block as a function of the local illumination variation between the current block template and the reference block template. The parameters of this function can be represented by the scale α and the offset β, forming a linear equation, i.e., To compensate for illumination variations, p[x] is the reference sample pointed to by the MV at position x on the reference image. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required except for signaling the LIC flag used for AMVP mode to indicate the use of LIC.

[0102] The local illumination compensation proposed in JVET-O0066 is used in ECM for single-prediction inter-frame CUs with the following modifications. • Intra-frame adjacent samples can be used in the LIC parameter export; • LIC is disabled for blocks with fewer than 32 luminance samples; • For both non-subblock and affine modes, the LIC parameter export is performed based on the template block sample corresponding to the current CU, rather than the partial template block sample corresponding to the first top-left 16x16 cell. • Samples of the reference block template are generated by using a MC with a block MV without rounding it to integer pixel precision.

[0103] Handling out-of-bounds (OOB) samples

[0104] In bidirectional motion compensation, out-of-bounds (OOB) prediction samples are discarded, and only non-OOB predictors are used to generate the final predictor when available. Specifically, and This indicates the position of a predicted sample within the current block. and (x=0,1) represents the MV of the current block; , , and These are the positions of the four boundaries of the image. A predicted sample is considered out-of-bounds (OOB) when at least one of the following conditions is met: in half_pixel An 8 represents the distance of a half-pixel sample in a 1 / 16 pixel sample precision. After checking the OOB conditions for each sample, the final predicted sample for a bidirectional block is generated as follows: if It is OOB and If it is not OOB, then ; Otherwise, if Is it not OOB and If it is OOB, then ; otherwise When BCW is enabled, the OOB check process is also available.

[0105] Finally, note that this sample-adaptive bidirectional prediction process is only applied to the prediction unit, for which at least the reference block is initially detected as out of bounds, either partially or entirely. Therefore, the block-level OOB criterion is checked first. If both prediction blocks are not OOB, then the usual bidirectional prediction occurs.

[0106] In-loop filter

[0107] There are a total of three in-loop filters in VVC. In addition to the deblocking filter and the SAO (two in-loop filters in HEVC), an adaptive loop filter (ALF) is applied. The ALF includes a luma ALF, a chroma ALF, and a cross-component ALF (CC-ALF). The ALF filtering process is designed so that the luma ALF, chroma ALF, and CC-ALF can be executed in parallel. The filtering order in VVC is deblocking filter, SAO, and ALF. The SAO in VVC is the same as the SAO in HEVC.

[0108] In VVC, an adaptive loop filter (ALF) with block-based filter adaptation is applied. For the luminance component, one of 25 filters is selected for each 4×4 block based on the direction and activity of the local gradient.

[0109] Two diamond-shaped filter shapes were used (e.g.) Figure 8 (As shown). A 7×7 rhombus is applied to the luminance component, and a 5×5 rhombus is applied to the chrominance component.

[0110] For the luminance component, each 4×4 block is classified into one of 25 classes. The classification index C is based on its directionality. and activity The quantized values ​​are derived as follows: In order to calculate and We first compute the gradients in the horizontal, vertical, and two diagonal directions using the 1-D Laplacian operator: Where index and This refers to the coordinates of the top-left sample within a 4x4 block, and Indicator at coordinates Reconstructed samples at the location.

[0111] To reduce the complexity of block classification, downsampling 1-D Laplacian operator computation is applied. As shown in Figure 9, the same downsampling position is used for gradient calculation in all directions.

[0112] Then, the gradients in the horizontal and vertical directions The maximum and minimum values ​​are set to: The maximum and minimum values ​​of the gradients in the two diagonal directions are set as follows: To derive directionality The values ​​are compared with each other and with two thresholds. and Comparison: Step 1. If and If both are true, then D is set to 0; Step 2. If If yes, continue from step 3; otherwise, continue from step 4. Step 3. If If the condition is met, then D is set to 2; otherwise, D is set to 1. Step 4. If If the value is 4, then D is set to 4; otherwise, D is set to 3.

[0113] Activity value The calculation is as follows: It is further quantized to include the range from 0 to 4, and the quantized value is represented as

[0114] No classification method was applied for the chromaticity components in the image.

[0115] Before filtering each 4×4 luminance block, geometric transformations (such as rotation or diagonal and vertical flips) are applied to the filter coefficients, depending on the gradient values ​​calculated for that block. and the corresponding filter clipping value. This is equivalent to applying these transformations to samples in the filter's support region. The idea is to make different blocks to which the ALF is applied more similar by aligning the directionality of the ALF.

[0116] Three geometric transformations are introduced: diagonal transformation, vertical flip, and rotation. diagonal: , Vertical Flip: , Rotation: . in It is the size of the filter and These are coefficient coordinates, which make the position... In the top left corner, and in position In the bottom right corner. Depending on the gradient values ​​calculated for that block, the transformation is applied to the filter coefficients. and clipping value The relationship between the transformation and the four gradients in the four directions is summarized in the table below.

[0117] On the decoder side, when ALF is enabled for CTB, each sample within the CU Filtering, resulting sample values As shown below: in Represents the decoding filter coefficients. It is a clipping function, and This represents the decoding clipping parameters. Variables k and l in... The values ​​vary between L and L, where L represents the filter length.

[0118] Clipping function This formula corresponds to the function Clipping introduces nonlinearity to make ALF more effective by reducing the influence of neighboring sample values ​​that are too different from the current sample value.

[0119] CC-ALF refines each chromaticity component by applying an adaptive, linear filter to the luminance channel and then using the output of that filter operation for chromaticity refinement, using luminance sample values. Figure 10(a) Provides a system-level schematic diagram of CC-ALF processing relative to SAO, luminance ALF, and chrominance ALF processing.

[0120] Filtering in CC-ALF involves using linear, diamond-shaped filters ( Figure 10 (b) This is implemented by applying it to the luminance channel. A filter is used for each chroma channel, and this operation is represented as follows: in It is the chromaticity component i, the position being refined. Based on The brightness position, It is the filter support region in the luminance component. Represents filter coefficients

[0121] like Figure 10 As shown, the luminance filter support is the region paired with the current chroma sample after considering the spatial scaling factor between the luminance and chroma planes.

[0122] In the VVC reference software, CC-ALF filter coefficients are calculated by minimizing the mean squared error of each chroma channel relative to the original chroma content. To achieve this, the VTM algorithm uses a coefficient derivation process similar to that used for chroma ALF. Specifically, the correlation matrix is ​​derived, and the coefficients are computed using a Cholesky decomposition solver to attempt to minimize the mean squared error metric. Up to eight CC-ALF filters can be designed and sent per image during filter design. The resulting filters are then instructed for each of the two chroma channels based on the CTU.

[0123] Adaptive in-loop filters (such as the ALF in VVC) perform luminance and chrominance filtering in parallel to reduce the mean square error (MSE) between the reconstructed and original samples. For more details, see... Figure 10 In VVC, ALF first filters both luminance and chrominance in parallel, and then applies cross-component filtering (CC-ALF) to further improve chrominance. Signaling the filter with coarse spatial granularity means that the filter changes at most at the CTU level.

[0124] CC-ALF in VVC uses luminance samples as input before ALF and therefore ignores any improvements derived from the luminance ALF. Even if CC-ALF were to use the luminance filtered by ALF as input, there would be little benefit, because CC-ALF uses a Wiener filter (which is signaled at the CTU level) to improve chroma, and at this scale, small changes to the input derived from the Wiener filter will not change the output much (i.e., the filter coefficients are slightly different, but the output is largely the same).

[0125] Instead of CTU-level filters, more localized (and independent of coding or prediction partitions) luma-to-chroma filters are needed to fully convert luma improvements into chroma improvements, but the signaling of such filters is prohibited by significant signaling costs.

[0126] To improve the use of the luminance output of the ALF for improved chroma reconstruction, an example embodiment of the present invention applies a filter that directly converts the luminance enhancement to the chroma enhancement before applying the chroma ALF or CC-ALF. These filters need to be derived and applied locally with fine spatial granularity (i.e., different filters for each 4x4 block). According to an example embodiment of the present invention, a convolutional cross-component model (CCCM) is used to map the enhanced luminance to the enhanced chroma without additional signaling. A variation of this approach has been described in [1], in which the CCCM is used to map the luminance residual to the chroma correction.

[0127] Before describing the exemplary embodiments disclosed herein in detail, refer to Figure 12 It illustrates a simplified block diagram of various electronic devices applicable to the practice of exemplary embodiments of the present invention.

[0128] Figure 12 A block diagram of a possible, non-limiting, exemplary system in which example embodiments can be practiced is shown. Figure 12 In the middle, User Equipment (UE) 10 and such Figure 12 Wireless communication is conducted via wireless network 1 or network 1. For example... Figure 12 The wireless network 1 or network 1 in the document may include a communication network, such as a mobile network, for example, mobile network 1 or a first mobile network as disclosed herein. In this document, as... Figure 12 Any reference to Wireless Network 1 in this document can be considered a reference to any wireless network as disclosed herein. Additionally, as... Figure 12The wireless network 1 may also include hardwired features that the communication network may require. The UE is wireless and is typically a mobile device that can access the wireless network. For example, the UE may be a mobile phone (or "cellular" phone) and / or a computer with mobile terminal capabilities. For example, the UE or mobile terminal may also be a portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile device that performs voice signaling and / or data exchange with the RAN.

[0129] UE 10 includes one or more processors DP 10A, one or more memories MEM 10B, and one or more transceivers TRANS 10D interconnected via one or more buses. Each transceiver in the one or more transceivers TRANS 10D includes a receiver and a transmitter. The one or more buses may be address, data, or control buses and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optic cable, or other optical communication device. The one or more transceivers TRANS 10D may optionally be connected to one or more antennas for communicating with NN 12 and ISM 13, respectively. The one or more memories MEM 10B include computer program code PROG 10C. UE 10 communicates with NN 12 and / or ISM 13 via wireless links 11 or 16.

[0130] NN 12 (NR / 5G Node B, evolved NB, or LTE equipment) is related to, for example, NR / 5G Node B, evolved NB, or LTE equipment. Figure 12The network node NN 12 communicates with devices such as ISM 13 and UE 10 (e.g., primary or secondary base stations, for NR or LTE LTE). NN 12 provides access to wireless devices (such as UE 10) to the wireless network 1. NN 12 includes one or more processors DP 12A, one or more memories MEM 12B, and one or more transceivers TRANS 12D interconnected via one or more buses. According to an example embodiment, these TRANS 12Ds may include X2 and / or Xn interfaces for performing the example embodiment. Each of the one or more transceivers TRANS 12Ds includes a receiver and a transmitter. The one or more transceivers TRANS 12Ds may optionally be connected to one or more antennas for communicating with UE 10 at least on link 11. One or more memories MEM 12B and computer program code PROG 12C are configured, together with one or more processors DP 12A, to cause NN 12 to perform one or more of the operations described herein. NN 12 can communicate with another gNB or eNB, or device (such as ISM 13), for example, via link 16. Additionally, link 11, link 16, and / or any other link can be wired, wireless, or both, and can implement, for example, an X2 or Xn interface. Furthermore, link 11 and / or link 16 can communicate via other network devices, such as, but not limited to, Figure 12 The NN 12 is an NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 device. It can perform the functions of MME (Mobility Management Entity) or SGW (Serving Gateway), such as user plane functions, and / or access management functions for LTE, as well as similar functions for 5G.

[0131] ISM 13 may be a wireless device for WiFi, Bluetooth, or associated with mobile functionality devices such as AMF or SMF. Additionally, ISM 13 may include an NR / 5G node B, or a possible evolved NB base station, such as a primary node base station or a secondary node base station (e.g., for NR or LTE LTE), which communicates with the device (such as NN 12 and / or UE 10 and / or Wireless Network 1). ISM 13 includes one or more processors DP 13A, one or more memories MEM 13B, one or more network interfaces, and one or more transceivers TRANS 13D interconnected via one or more buses. According to an example embodiment, these network interfaces of ISM 13 may include X2 and / or Xn interfaces for performing the example embodiment. Each transceiver in the one or more transceivers TRANS 13D includes a receiver and a transmitter, which may optionally be connected to one or more antennas. The one or more memories MEM 13B include computer program code PROG 13C. For example, one or more memory MEM 13B and computer program code PROG 13C are configured, together with one or more processors DP 13A, to cause ISM 13 to perform one or more of the operations described herein. ISM 13 can communicate with another mobility function device and / or eNB (such as NN 12 and UE 10) or any other device using, for example, link 11, or link 16 or another link. Figure 12 Link 16 shown can be used to communicate with NN 12. These links can be wired, wireless, or both, and can implement, for example, an X2 or Xn interface. Additionally, as mentioned above, links 11 and / or 16 can be connected via other network devices, such as, but not limited to, NCE / MME / SGW devices, such as… Figure 12 NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14.

[0132] Figure 12 One or more buses of the device can be address, data, or control buses, and can include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optic or other optical communication devices, wireless channels, etc. For example, one or more transceivers TRANS 12D, TRANS 13D, and / or TRANS 10D can be implemented as remote radio heads (RRHs), other components of the NN 12 are physically located at a different location from the RRH, and these devices can include one or more buses, which can be partially implemented as fiber optic cables to connect other components of the NN 12 to the RRH.

[0133] Note that, although Figure 12Network nodes such as NN 12 and ISM 13 are shown, but any of these nodes can be incorporated into or be incorporated into an eNodeB, or eNB or gNB (such as for LTE and NR), and will still be configured to perform the example implementation.

[0134] It should also be noted that the description herein refers to a "cell" performing functions; however, it should be clear that a gNB that forms a cell and / or user equipment and / or mobility management function equipment will perform these functions. Furthermore, cells constitute a part of a gNB, and each gNB may have multiple cells.

[0135] Wireless Network 1 or any network that it may represent may or may not include NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14, which may include (NCE) network control element functions, MME (Mobility Management Entity) / SGW (Serving Gateway) functions, and / or Serving Gateway (SGW), and / or MME (Mobility Management Entity) and / or SGW (Serving Gateway) functions, and / or User Data Management (UDM) functions, and / or PCF (Policy Control) functions, and / or Access and Mobility Management (AMF) functions, and / or Session Management (SMF) functions, and / or Location Management (LMF) functions, and / or Authentication Server (AUSF) functions, as well as functions that provide connectivity to other networks (such as telephone networks and / or data communication networks (e.g., the Internet)), and functions configured to perform any 5G and / or NR operations other than or in lieu of other standard operations at the time of this application. NRNCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 can be configured to perform operations of exemplary embodiments of communication technologies based on LTE, NR, 5G, and / or any standards, as performed or discussed at the time of this application. Furthermore, it should be noted that operations according to exemplary embodiments performed by NN 12 and / or ISM 13 can also be performed at NRNCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14.

[0136] NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 includes one or more processors DP 14A interconnected via one or more buses coupled to link 13 and / or link 16, one or more memory MEM 14B, and one or more network interfaces (N / WI / F). According to an example embodiment, these network interfaces may include X2 and / or Xn interfaces for performing the example embodiment. The one or more memory MEM 14B includes computer program code PROG 14C. The one or more memory MEM 14B and computer program code PROG 14C are configured, together with the one or more processors DP 14A, to cause NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 to perform one or more operations that may be required for supporting operations according to the example embodiment.

[0137] Note that NN 12 and / or ISM 13 and / or UE 10 can be configured (e.g., based on standard implementations, etc.) to perform Location Management Function (LMF) functionality. LMF functionality can be implemented in any of these network devices, or other devices associated with them. Furthermore, LMF, such as... Figure 12 The LMF of MME / SGW / UDM / PCF / AMF / SMF / LMF 14, at least as described below, can be located in the same position as UE10 so as to be compatible with... Figure 12 The NN 12 and / or ISM 13 are separated for performing operations according to the example embodiments disclosed herein.

[0138] Wireless Network 1 can implement network virtualization, which is the process of combining hardware and software network resources, as well as network functions, into a single, software-based managed entity (virtual network). Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as external, combining many networks or parts of networks into virtual units, or internal, providing network-like functionality to software containers on a single system. Note that the virtualized entities resulting from network virtualization are still implemented at some level using hardware (such as processors DP 10, DP 12A, DP 13A, and / or DP 14A, and memory MEM 10B, MEM12B, MEM 13B, and / or MEM 14B), and these virtualized entities also create technical effects.

[0139] Computer-readable storage devices MEM 12B, MEM 13B, and MEM 14B can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, magnetic storage devices and systems, optical storage devices and systems, fixed memory, and removable memory. Computer-readable storage devices MEM 12B, MEM 13B, and MEM 14B can be components for performing storage functions. As a non-limiting example, processors DP 10, DP 12A, DP 13A, and DP 14A can be of any type suitable for the local technical environment and can include one or more of the following: general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), and processors based on multi-core processor architectures. Processors DP 10, DP 12A, DP 13A, and DP 14A can be components for performing functions such as controlling UE 10, NN 12, ISM 13, and other functions as described herein.

[0140] Typically, various embodiments of any of these devices may include, but are not limited to, cellular phones such as smartphones, tablets, personal digital assistants (PDAs) with wireless communication capabilities, portable computers with wireless communication capabilities, image capture devices (such as digital cameras) with wireless communication capabilities, gaming devices with wireless communication capabilities, music storage and playback devices with wireless communication capabilities, internet devices that allow wireless internet access and browsing, tablets with wireless communication capabilities, and portable units or terminals incorporating combinations of such functions.

[0141] Additionally, various embodiments of any of these devices can be used with any other node of this type associated with a UE vehicle, an aerial platform station, or a ground network, or any type of unmanned radio, or a radio in an aircraft, or other airborne vehicle, or a vessel (such as a ship) traveling on water.

[0142] Examples of the present invention provide at least an improved ALF pipeline that uses CCCM to locally map the improved luminance of the ALF to the improved chrominance. The output of the CCCM filtering stage is then used as input to both the ALF chrominance and CC-ALF. Examples of the present invention also provide several embodiments to address more specific situations.

[0143] In VVC, ALF uses Wiener filters to reduce the MSE of luminance and chrominance reconstruction. Filter coefficients are selected from a fixed set of coefficients using signaled indices, or explicitly signaled from the encoder to the decoder. Signaling is involved in both cases. See also Figure 10A cross-component variant called CC-ALF improves chromaticity by applying a Wiener filter to the luminance component to obtain a correction term for the chromaticity component. In all ALF-related filter derivations, the goal is (at the encoder) to minimize the error relative to the original luminance and chromaticity.

[0144] The ALF pipeline in the VVC on the decoder side is summarized in the following steps: 1a. Apply Luminance ALF to the reconstructed luminance; 2a. Apply Chromaticity ALF to the reconstructed chromaticity; 3a. Apply CC-ALF to the reconstructed luminance (i.e., the input of step 1a) to obtain chromaticity correction; 4a. Add the output of step 3a to the output of step 2a.

[0145] In an exemplary embodiment of the present invention, the improved ALF pipeline is as follows: Figure 11 The example shown is introduced.

[0146] Figure 11 An improved ALF pipeline with CCCM-based chromaticity updates is shown.

[0147] In the illustrated embodiment, the luminance input to the CC-ALF is the output of the ALF luminance stage, but it can also be configured to be the output of the SAO luminance stage. Additionally, depending on the embodiment, the CCCM model export can be placed before the SAO stage. Optional input paths to the CCCM filtering stage are used when some output samples from the CCCM filtering stage are mixed with SAO output samples, or when some CCCM filtering output samples are exactly SAO output samples. It should be understood that all illustrated CCCM stages can be replaced with any cross-component prediction tool (such as CCLM, GL-CCCM), or any other cross-component prediction tool, if desired.

[0148] More importantly, the additional filtering stage is inserted between the ALF luminance and ALF chrominance stages.

[0149] This filtering stage is based on the CCCM method and therefore does not require signaling, and uses a different minimization objective than ALF.

[0150] In CCCM, the minimization objective is not the original chroma, but chroma reconstruction and model derivation are also performed at the decoder. The following steps describe the improved pipeline on the decoder side. 1b. Using reconstructed luminance and chrominance as reference samples, obtain the CCCM filter at high spatial granularity; 2b. Apply Luminance ALF to the reconstructed luminance; 3b. Using the output of step 2b as input, apply the CCCM filter from step 1b; 4b. Apply chromatic ALF to step 3b; 5b. Apply CC-ALF to the output of either a) reconstructed luminance or b) step 2b to obtain chromaticity correction; 6b. Add the output of step 5b to the output of step 4b.

[0151] The output of step 2b is luminance, which has a higher quality than the luminance used as input in step 1b. When the higher quality luminance is used as input in step 3b, essentially a higher quality version of chroma is obtained as the output according to an exemplary embodiment of the invention. The CCCM filter resides in the color space characteristics (obtained by the model in step 1b), but converts luminance corrections (such as corrected edges / gradients or smoothed / sharpened textures) into a higher quality version of chroma.

[0152] One of the main benefits of the improved ALF pipeline is its granularity, at which CCCM model derivation and filtering occur. The CCCM model is derived and applied to small blocks that can be independent of the encoding and prediction partitions. For example, the blocks can be 1x1, 2x2, 4x4, etc., and can have square or rectangular shapes. These blocks can also be overlapping or distinct. Compared to CTU-level CC-ALF filters, the proposed method can track the cross-component model with high spatial accuracy and thus also maps fine-grained luma improvements to chroma improvements. This contrasts with CC-ALF, where filter coefficients are changed only at the CTU level, and only 8 different filters are available per image. With CCCM, thousands of filters can exist, for example, one filter per 8x8 block without any additional signaling costs.

[0153] In the proposed improved pipeline, CC-ALF can still be applied and still provides benefits because the CC-ALF filter is derived at the encoder by minimizing the squared difference relative to the original sample.

[0154] In step 5b above, depending on the embodiment, the input to CC-ALF can be unfiltered reconstructed luminance (i.e., the input to step 1b) or ALF-filtered luminance (i.e., the output of step 2b).

[0155] In the filtering phase of step 3b, CCCM model performance can be taken into account. During model derivation, the model's MSE is acquired, and if the MSE is considered high, the CCCM phase can be skipped for a given block.

[0156] It should be understood that, if necessary, all mentioned CCCM stages can be replaced by any cross-component prediction tool (such as CCLM, GL-CCCM), or any other cross-component prediction tool. For clarity, the prior art cross-component prediction tool CCCM is used as an example in the following embodiments.

[0157] In the embodiments, CCCM model export and filtering can be performed at any given granularity, for example, in blocks of 1x1, 2x2, 4x4, 8x8, 16x16, etc., or using rectangular blocks such as 4x8 or 8x4.

[0158] In an embodiment, the CCCM model can be derived and applied at blocks, which can be different or overlapping.

[0159] In embodiments, CCCM blocks can be independent of or dependent on encoding or prediction partitions. For example, instead of using very fine granularity, blocks can follow chroma or luma partitions with some precision to accelerate model derivation and filtering.

[0160] In this embodiment, during the CCCM model export with MSE, the model can be checked at each block, and if the MSE exceeds a given threshold, the CCCM phase for a given block can be skipped.

[0161] In this embodiment, based on the foregoing embodiments, the MSE threshold may be fixed or may be signaled from the encoder to the decoder.

[0162] In this embodiment, based on the foregoing embodiments, the MSE threshold can be inferred based on the base coding partition, luminance sample value, or chrominance sample value.

[0163] In the embodiments, the CCCM model can have any number of filter coefficients.

[0164] In the embodiments, CCCM model derivation and filtering can be replaced by simplified variants such as CCLM.

[0165] In this embodiment, CCCM model derivation and filtering can be replaced by more advanced variants, such as variants that take into account gradient and location information (such as GL-CCCM).

[0166] In this embodiment, the CCCM model export stage can be placed before or after the SAO stage.

[0167] In this embodiment, the CCCM model export stage can be placed at any point before the luminance ALF stage.

[0168] In this embodiment, the luminance input of the CC-ALF stage can be the input of the ALF luminance stage or the output of the ALF luminance stage.

[0169] In an embodiment, the filter may consist of auxiliary information to guide it in a way that better improves regions or samples with certain characteristics. Examples of such auxiliary information are as follows: • The filter input can include residual information from the luma block; • The filter input can include prediction information for the luminance blocks; • The filter input may include one or more transform coefficients from the transform coefficients of the luminance block; • Filter inputs may include the inputs and / or outputs of earlier filtering operations. For example, the inputs and / or outputs of a deblocking filter, an SAP filter, or an ALF filter; • Filter inputs can include the difference between the input and output of earlier filtering operations. Examples include the difference between the input and output of a deblocking filter, an SAP filter, and an ALF filter. Scaling operations can also be applied to the difference before it is fed into the CCCM filter.

[0170] In this embodiment, the use of the CCCM stage can signal from the encoder to the decoder.

[0171] In embodiments, the use of filters can be signaled at different granularities; for example, signaling can be performed on a per-filtering-unit basis (e.g., 1x1, 2x2). Alternatively, signaling can be performed at a granularity different from the filtering granularity. For example, filter activation can be performed at a coarser granularity for each CTU, and filtering can be performed at a finer granularity (e.g., 1x1, 2x2, 4x4, ...) when the filtering mode is enabled for a CTU. In this case, on the encoder side, the decision on whether to use a filter can be made for the entire CTU based on distortion rate optimization.

[0172] In embodiments, different types and / or sizes of filters (such as conventional CCCM, GL-CCCM, CCLM, or any other variant) can be determined for each CTU, slice, word picture, image, or sequence level, and signaled accordingly.

[0173] In an embodiment, the CCCM stage can be applied to all chromaticity components or a specific set of chromaticity components.

[0174] In the embodiments, the proposed ALF pipeline can be used in any color space.

[0175] In an embodiment, the additional chromaticity to chromaticity CCCM stage can be inserted between the ALF Cb and ALF Cr stages, thus making the chromaticity ALF also sequential.

[0176] In this embodiment, the CCCM model export and filtering stages may use downsampled and / or raw luminance samples.

[0177] In this embodiment, the filtering units can be composed of non-overlapping blocks. In this case, the filter derivation process for all filtering units can be performed in parallel. Furthermore, after the parameters of a certain unit are derived, CCCM filtering can begin with minimal delay in the pipeline.

[0178] Figure 13 It shows that it can be made by, but is not limited to, devices (e.g., Figure 12 The operations performed by the device (UE 10) in the example. Figure 13 As shown in step 1310, there exists a method for obtaining at least one convolutional cross-component model filter for the sample set. Figure 13 As shown in step 1320, the sample set is a reconstruction sample for the two channels of the image. Figure 13 As shown in step 1330, there exists a reconstructed sample set for applying at least one filter to the first channel. Figure 13 As shown in step 1340, there exists a reconstructed sample set for applying at least one filter to the first channel. Figure 13 As shown in step 1350, there exists a filter whose output is used as input to the reconstructed sample set of the first channel, and at least one convolutional cross-component model filter is applied; as Figure 13 As shown in step 1360, there exists an output from applying at least one filter to at least one convolutional cross-component model filter. Figure 13 As shown in step 1370, there is a cross-component filter applied to the reconstructed sample set of the first channel, or applied to the output of at least one filter for the reconstructed sample set of the first channel, to obtain a correction for the output of at least one filter using the output of at least one convolutional cross-component model filter as input.

[0179] According to the example embodiments described in the above paragraphs, there exists a method that uses the output of at least one convolutional cross-component model filter as input, and adds the output of the cross-component filter to the output of at least one filter.

[0180] According to the example embodiments described in the above paragraphs, the convolutional cross-component model filter is derived and applied to blocks that are not determined by any coding or prediction partitions.

[0181] According to the example embodiments described in the preceding paragraphs, the blocks have a square or rectangular shape, and the blocks may be overlapping or distinct. The use of the convolutional cross-component model for each block is determined without additional signaling.

[0182] A stored procedure code (such as) Figure 12 Non-transitory computer-readable media (such as RPOG 10C) Figure 12 (MEM 12B in the text), the program code is generated by at least one processor (such as...) Figure 12 DP 10A and / or DP 10F) are executed to perform at least the operations described in the above paragraphs.

[0183] According to an exemplary embodiment of the present invention as described above, there exists an apparatus comprising: having an input for using the output of at least one convolutional cross-component model filter as input, and adding (e.g.) the output of the cross-component filter. Figure 12 A component that includes one or more transceivers 10D and / or one or more transceivers 13D; MEM 10B and / or MEM 13B; PROG 10C and / or PROG 13C; and DP 10A and / or DP 13A) to the output of at least one filter, wherein a convolutional cross-component model filter is derived and applied (as shown in the image). Figure 12 One or more transceivers 10D and / or one or more transceivers 13D; MEM 10B and / or MEM 13B; PROG 10C and / or PROG 13C; and DP 10A and / or DP13A) are used in blocks not determined by any encoding or prediction partitions, wherein the blocks have a square or rectangular shape, and wherein the blocks may overlap or be different, wherein the use of the convolutional cross-component model for each block (as shown in the figure) Figure 12 One or more transceivers 10D and / or one or more transceivers 13D; MEM 10B and / or MEM 13B; PROG 10C and / or PROG 13C; and DP 10A and / or DP 13A) are determined without additional signaling.

[0184] In an exemplary aspect of the invention according to the foregoing paragraphs, wherein at least the components for adding, exporting, applying, and determining include: via a process that can be handled by at least one processor [e.g. Figure 12 The computer program executed by DP 10A and / or DP 13A in [e.g.] Figure 12 Non-transitory computer-readable media encoded with PROG 10C and / or PRPG 13C [e.g.] Figure 12 MEM10B and / or MEM 13B in the MEM10B and / or MEM 13B.

[0185] Furthermore, according to exemplary embodiments of the present invention, there exists a circuit system for performing operations according to exemplary embodiments of the invention disclosed herein. This circuit system may include any type of circuit system, including content encoding circuit systems, content decoding circuit systems, processing circuit systems, image generation circuit systems, data analysis circuit systems, etc. Additionally, this circuit system may include discrete circuit systems, application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs), processors specifically configured by software to perform corresponding functions, or dual-core processors having software and corresponding digital signal processors, etc. Furthermore, necessary outputs to and from the circuit system, functions performed by the circuit system, and interconnections (possibly via inputs and outputs) between the circuit system and other components that may include other circuit systems are provided to perform exemplary embodiments of the invention as described herein.

[0186] According to the exemplary embodiments of the present invention disclosed in this application, the provided "circuit system" may include at least one or more, or all of the following: (a) Hardware circuit implementation only (such as implementation of analog and / or digital circuit systems only); (b) A combination of hardware circuitry and software, such as (if available): (i) A combination of analog and / or (multiple) digital hardware circuits with software / firmware; and (ii) Any portion of the hardware processor(s) having software (including the digital signal processor(s)), software, and memory(s), which work together to enable a device (such as a mobile phone or server) to perform various functions (such as functions or operations according to exemplary embodiments of the invention disclosed herein); and (c) (multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or a portion thereof, which require software (e.g., firmware) to operate, but may be absent when operation is not required.

[0187] According to exemplary embodiments of the present invention, there exists a sufficient circuit system for at least performing the novel operations of exemplary embodiments of the invention disclosed herein, whereby "circuit system" as used herein refers to at least the following: (a) Hardware circuit implementation only (such as implementation of analog and / or digital circuit systems only); and (b) A combination of hardware circuitry and software (and / or firmware), such as (if available): (i) a combination of (multiple) processors, or (ii) a portion of (multiple) processors / software (including (multiple) digital signal processors), software, and (multiple) memories, which work together to enable a device (such as a mobile phone or server) to perform various functions; and (c) A circuit, such as (a plurality of) microprocessors or part of (a plurality of) microprocessors, that requires software or firmware to operate, even if the software or firmware does not exist physically.

[0188] This definition of 'circuit system' applies to all uses of the term in this application (including in any claim). As another example, as used in this application, the term "circuit system" will also cover only the implementation of a processor (or processors) or a portion thereof and its accompanying software and / or firmware. For example, and if applicable to a particular claim element, the term "circuit system" will also cover baseband integrated circuits or application processor integrated circuits for mobile phones, or similar integrated circuits in servers, cellular network devices, or other network devices.

[0189] Typically, the various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while others may be implemented in firmware or software, which may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. While various aspects of the invention may be shown and described as block diagrams, flowcharts, or other graphical representations, it is well understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or combinations thereof.

[0190] Embodiments of the present invention can be practiced in various components such as integrated circuit modules. The design of integrated circuits is largely a highly automated process. Complex and powerful software tools can be used to transform logic-level designs into semiconductor circuit designs ready for etching and formation on semiconductor substrates.

[0191] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. All embodiments described in this detailed description are exemplary embodiments provided to enable those skilled in the art to make or use the invention, and are not intended to limit the scope of the invention as defined by the claims.

[0192] The foregoing description has provided a complete and informative description of the best methods and apparatus currently contemplated by the inventors for carrying out the invention, by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the art when read in conjunction with the accompanying drawings and appended claims, given the foregoing description. Nevertheless, all such modifications and similar alterations to the teachings of the exemplary embodiments of the invention will still fall within the scope of the invention.

[0193] It should be noted that the terms “connection,” “coupling,” or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may encompass the presence of one or more intermediate elements between two elements that are “connected” or “coupled” together. The coupling or connection between elements can be physical, logical, or a combination thereof. As adopted herein, as several non-limiting and non-exhaustive examples, two elements may be considered to be “connected” or “coupled” together by using one or more wires, cables, and / or printed electrical connections, and by using electromagnetic energy (such as electromagnetic energy having wavelengths in the radio frequency region, microwave region, and optical (both visible and invisible) regions).

[0194] Furthermore, some features of the preferred embodiments of the invention can be used advantageously without the need for corresponding use of other features. Therefore, the foregoing description should be considered merely as an illustration of the principles of the invention, and not as a limitation thereof.

Claims

1. An apparatus comprising: At least one processor; as well as At least one non-transitory memory storing instructions, which, when executed by the at least one processor, cause the device to at least: For each sample set, obtain at least one convolutional cross-component model filter. The sample set mentioned above is a reconstructed sample for two channels of the image; At least one filter is applied to the reconstructed sample set of the first channel; Using the output of at least one filter of the reconstructed sample set for the first channel as input, the at least one convolutional cross-component model filter is applied; Apply at least one filter to the output of the at least one convolutional cross-component model filter; as well as The cross-component filter is applied to the reconstructed sample set of the first channel, or to the output of the at least one filter for the reconstructed sample set of the first channel, to obtain a correction for the output of the at least one filter using the output of the at least one convolutional cross-component model filter as input.

2. The apparatus of claim 1, wherein the at least one non-transitory memory stores instructions, the instructions being executed by the at least one processor, such that the apparatus: Using the output of the at least one convolutional cross-component model filter as input, the output of the cross-component filter is added to the output of the at least one filter.

3. The apparatus of claim 1, wherein the convolutional cross-component model filter is derived and applied to blocks not determined by any coding or prediction partitions.

4. The apparatus of claim 3, wherein the blocks have a square or rectangular shape, and wherein the blocks can be overlapping or different.

5. The apparatus of claim 3, wherein the use of the convolutional cross-component model for each block is determined without additional signaling.

6. A method comprising: For each sample set, obtain at least one convolutional cross-component model filter. The sample set mentioned above is a reconstructed sample for two channels of the image; At least one filter is applied to the reconstructed sample set of the first channel; Using the output of at least one filter of the reconstructed sample set for the first channel as input, the at least one convolutional cross-component model filter is applied; Apply at least one filter to the output of the at least one convolutional cross-component model filter; as well as The cross-component filter is applied to the reconstructed sample set of the first channel, or to the output of the at least one filter for the reconstructed sample set of the first channel, to obtain a correction for the output of the at least one filter using the output of the at least one convolutional cross-component model filter as input.

7. The apparatus of claim 6, wherein the at least one non-transitory memory stores instructions, the instructions being executed by the at least one processor, such that the apparatus: Using the output of the at least one convolutional cross-component model filter as input, the output of the cross-component filter is added to the output of the at least one filter.

8. The method of claim 6, wherein the convolutional cross-component model filter is derived and applied to blocks not determined by any encoding or prediction partitions.

9. The method of claim 8, wherein the blocks have a square or rectangular shape, and wherein the blocks can be overlapping or different.

10. The method of claim 8, wherein the use of the convolutional cross-component model for each block is determined without additional signaling.

Images