Smoothing filtered chroma reconstruction samples as an additional input to cross-component ALF or chroma ALF in-loop

Abstract

Additional in-loop filtering can be using Gausian smoothing, anisotropic nonlinear smoothing, and / or enhancement. Two ideas are proposed. As an additional input to CC-ALF use filtered Chroma reconstruction Chroma buffers as an additional input to CC-ALF. New additional filter coefficients are trained by encoder and are signaled to the bit stream. From the decoder side signaled coefficients are read and CC-ALF applied; or As an additional input to Chroma ALF use filtered Chroma reconstruction Chroma buffers as an additional input to Chroma ALF. New additional filter coefficients are trained by encoder and are signaled to the bit stream. From the decoder side signaled coefficients are read and Chroma ALF applied.

Description

FILTERED CHROMA RECONSTRUCTION BUFFERS AS AN ADDITIONALINPUT TO CROSS-COMPONENT ACTIVE LOOP FILTERING AND / OR CHROMA ACTIVE LOOP FILTERING TECHNICAL FIELD:

[0001] The teachings in accordance with the exemplary embodiments of thisinvention relate generally to using filtered Chroma reconstruction Chroma buffers asan additional input to CC-ALF and, more specifically, relate to using filtered Chromareconstruction Chroma buffers as an additional input to CC-ALF, wherein a newadditional filter coefficients may be trained by an encoder and signaled to the bit stream.BACKGROUND:

[0002] This section is intended to provide a background or context to theinvention that is recited in the claims. The description herein may include concepts thatcould be pursued, but are not necessarily ones that have been previously conceived orpursued. Therefore, unless otherwise indicated herein, what is described in this sectionis not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.

[0003] Certain abbreviations that may be found in the description and / or in theFigures are herewith defined as follows:VVC – Versatile Video CodingECM – Enhanced Compression ModelALF – Adaptive Loop FilterCC-ALF - Cross-component Adaptive Loop FilterMSE – Mean Square ErrorAPS – Adaptive Parameter SetRDO – Rate Distortion Optimization

[0004] Hybrid video codecs, for example ITU-T H.263, H.264 / AVC andHEVC, may encode the video information in two phases. At first, pixel values in a certain picture are (or “block”) are predicted for example by motion compensationmeans (finding and indicating an area in one of the previously coded video frames thatcorresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). In the first phase, predictive coding may be applied, for example, as so-called sample prediction and / or so-called syntax prediction.

[0005] In the sample prediction, pixel or sample values in a certain picture areaor "block" are predicted. These pixel or sample values can be predicted, for example, using one or more of motion compensation or intra prediction mechanisms.

[0006] Example embodiments of the invention work to improve at least theoperations as stated above. SUMMARY:

[0007] This section contains examples of possible implementations and is notmeant to be limiting.

[0008] In another example aspect of the invention, there is an apparatus, suchas a network side apparatus, comprising: at least one processor; and at least one non-transitory memory storing instructions, that when executed by the at least oneprocessor, cause the apparatus at least to: apply a cross component adaptive loop filteron a reconstructed luma to obtain a reconstructed Chroma; apply a Chroma adaptiveloop filter on the reconstructed Chroma to obtain a Chroma correction; and use theChroma correction as an additional input to improve a cross-component adaptive loopfilter for a video or image stream.

[0009] In still another example aspect of the invention, there is a method,comprising: applying a cross component adaptive loop filter on a reconstructed luma toobtain a reconstructed Chroma; applying a Chroma adaptive loop filter on thereconstructed Chroma to obtain a Chroma correction; and using the Chroma correctionas an additional input to improve a cross-component adaptive loop filter for a video or image stream.

[0010] A further example embodiment is an apparatus and a method comprisingthe apparatus and the method of the previous paragraphs, wherein the Chromacorrection is used as as an additional input to the cross component adaptive loop filterto derive filtered chroma reconstruction buffers, wherein the filtered chroma reconstruction buffers are used for the additional input to the cross component adaptive loop filter, wherein the cross-component adaptive loop filter is applying a Wiener filterto a luma component to correct at least one of a chroma component or the lumacomponent, wherein the cross-component adaptive loop filter comprises at least one ofan adaptive loop filter luma or an adaptive loop filter Chroma, and / or wherein the additional input is using the adaptive loop filter Chroma.

[0011] Wherein an additional filter shape is used to increase at least one of anumber of filter coefficients or clipping values trained as an additional input buffer forthe additional input, wherein at least one of the increased filter coefficients or clippingvalues are used on the video or image stream during the additional input to improve across-component adaptive loop filter, wherein the filtered Chroma reconstructionChroma buffers comprise a filtered Chroma, wherein the filtered Chroma is based onisotropic smoothing comprising at least one of Gaussian smoothing, anisotropicnonlinear smoothing, or enhancing, and / or wherein the cross-component adaptive loopfilter comprises luma filtering and Chroma filtering.

[0012] A non-transitory computer-readable medium storing program code, theprogram code executed by at least one processor to perform at least the method as described in the paragraphs above.

[0013] In yet another example aspect of the invention, there is an apparatuscomprising: means for applying a cross component adaptive loop filter on areconstructed luma to obtain a reconstructed Chroma; means for applying a Chromaadaptive loop filter on the reconstructed Chroma to obtain a Chroma correction; andmeans for using the Chroma correction as an additional input to improve a cross-component adaptive loop filter for a video or image stream.

[0014] In accordance with the example embodiments as described in theparagraph above, at least the means for applying and using comprises a networkinterface, and computer program code stored on a computer-readable medium and executed by at least one processor.

[0015] A communication system comprising the network side apparatus and theuser equipment side apparatus performing operations as described above. BRIEF DESCRIPTION OF THE DRAWINGS:

[0016] The above and other aspects, features, and benefits of variousembodiments of the present disclosure will become more fully apparent from the following detailed description with reference to the accompanying drawings, in which like reference signs are used to designate like or equivalent elements. The drawings are illustrated for facilitating better understanding of the embodiments of the disclosure and are not necessarily drawn to scale, in which:

[0017] FIG. 1A shows an example of the location of the left and above samplesand the sample of the current block involved in the CCLM mode;

[0018] FIG. 1B shows a Table 1 of Derivation of Chroma prediction mode fromluma mode when cclm_is enabled;

[0019] FIG. 1C shows a Table 2 Unified binarization table for Chromaprediction mode;

[0020] FIG. 1D shows a Table Mapping of the gradient calculated for one blockand the transformations;

[0021] FIG. 1E shows a Table of mappings of the gradient calculated for oneblock and the transformations;

[0022] FIG. 2A and FIG. 2B show reconstructed neighbouring samplesclassified into two classes using a threshold, which is the average of the lumareconstructed neighbouring samples;

[0023] FIG. 3 shows locations of the samples used for the derivation of CCCMfilter when six reference lines are used, in accordance with example embodiments ofthe invention;

[0024] FIG. 4 shows from left: 3-tap vertical, 3-tap horizontal, 5-tap cross, 25-tap diamond;

[0025] FIG. 5 shows an example of four reference lines neighboring to aprediction block;

[0026] FIG. 6 shows a Matrix weighted intra prediction process;

[0027] FIG. 7 shows Top and left neighboring blocks used in CIIP weightderivation;

[0028] FIG. 8 shows ALF filter shapes (Chroma: 5×5 diamond, luma: 7×7diamond);

[0029] FIG. 9 shows Subsampled Laplacian calculation;

[0030] FIG. 10 shows (a) Placement of CC-ALF with respect to other loopfilters (b) Diamond shaped filter;

[0031] FIG. 11 shows Improved CC-ALF pipeline with additional smoothedChroma buffers, in accordance with example embodiments of the invention;

[0032] FIG. 12 shows Improved Chroma ALF pipeline with additionalsmoothed Chroma buffers;

[0033] FIG. 13 shows a high level block diagram of various devices used incarrying out various aspects of the invention; and

[0034] FIG. 14 shows a method in accordance with example embodiments ofthe invention which may be performed by an apparatus. DETAILED DESCRIPTION:

[0035] In example embodiments of this invention there is proposed at least amethod and apparatus to use filtering or smoothed stage Chroma reconstructionChroma buffers as an additional input to CC-ALF, wherein a new additional filtercoefficients may be trained by an encoder and signaled to the bit stream.

[0036] As similarly stated above, hybrid video codecs, for example ITU-TH.263, H.264 / AVC and HEVC, may encode the video information in two phases. At first, pixel values in a certain picture are (or “block”) are predicted for example by motion compensation means. Such means as finding and indicating an area in one ofthe previously coded video frames that corresponds closely to the block being coded orby spatial means such as using the pixel values around the block to be coded in a specified manner. In the first phase, predictive coding may be applied, for example, as so-called sample prediction and / or so-called syntax prediction.

[0037] In the sample prediction, pixel or sample values in a certain picture areaor "block" are predicted. These pixel or sample values can be predicted, for example, using one or more of motion compensation or intra prediction mechanisms.

[0038] Motion compensation mechanisms (which may also be referred to asinter prediction, temporal prediction or motion-compensated temporal prediction or motion-compensated prediction or MCP) involve finding and indicating an area in oneof the previously encoded video frames that corresponds closely to the block being coded. Inter prediction may reduce temporal redundancy.

[0039] Intra prediction, where pixel or sample values can be predicted by spatialmechanisms, involve finding and indicating a spatial region relationship. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, i.e., eithersample values or transform coefficients can be predicted. Intra prediction is typicallyexploited in intra coding, where no inter prediction is applied.

[0040] In the syntax prediction, which may also be referred to as parameterprediction, syntax elements and / or syntax element values and / or variables derived from syntax elements are predicted from syntax elements (de)coded earlier and / or variables derived earlier. Non-limiting examples of syntax prediction are provided below.

[0041] In motion vector prediction, motion vectors e.g. for inter and / or inter-view prediction may be coded differentially with respect to a block-specific predicted motion vector. In many video codecs, the predicted motion vectors are created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions, sometimes referred to as advanced motion vector prediction (AMVP), is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signalling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or co-located blocks in temporal reference picture. Differential coding of motion vectors is typically disabled across slice boundaries. The block partitioning, e.g. from CTU to CUs and down to PUs, may be predicted.

[0042] In filter parameter prediction, the filtering parameters e.g. for sampleadaptive offset may be predicted. Prediction approaches using image information from a previously coded image can also be called as inter prediction methods which may also be referred to as temporal prediction and motion compensation. Prediction approachesusing image information within the same image can also be called as intra prediction methods.

[0043] Secondly, the prediction error, i.e. the difference between the predictedblock of pixels and the original block of pixels, is coded. This may be done by transforming the difference in pixel values using a specified transform (e.g. Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients.

[0044] By varying the fidelity of the quantization process, encoder can controlthe balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size of transmission bitrate).

[0045] In many video codecs, including H.264 / AVC and HEVC, motioninformation is indicated by motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder) or decoded (at the decoder) and the prediction source block in one of the previously coded or decoded images (or pictures). H.264 / AVC and HEVC, as many other video compression standards, a picture is divided into a mesh of rectangles, for each of which a similar block in one of the reference pictures is indicated for inter prediction. The location of the prediction block is coded as a motion vector that indicates the position of the prediction block relative to the block being coded.

[0046] In under developing Versatile Video Codec (VVC), there are newcoding tools. Such coding tools can be as follows: ^Intra prediction– 67 intra mode with wide angles mode extension,– Block size and mode dependent 4 tap interpolation filter,– Position dependent intra prediction combination (PDPC),– Cross component linear model intra prediction (CCLM),– Multi-reference line intra prediction,– Intra sub-partitions,– Weighted intra prediction with matrix multiplication;^ Inter-picture prediction– Block motion copy with spatial, temporal, history-based, and pairwiseaverage merging candidates, –Affine motion inter prediction,– sub-block based temporal motion vector prediction,– Adaptive motion vector resolution,– 8x8 block-based motion compression for temporal motion prediction– High precision (1 / 16 pel) motion vector storage and motioncompensation with 8-tap interpolation filter for luma component and 4- tap interpolation filter for Chroma component,– Triangular partitions,– Combined intra and inter prediction,– Merge with MVD (MMVD),– Symmetrical MVD coding,– Bi-directional optical flow,– Decoder side motion vector refinement,– Bi-prediction with CU-level weight;^ Transform, quantization and coefficients coding– Multiple primary transform selection with DCT2, DST7 and DCT8,– Secondary transform for low frequency zone,– Sub-block transform for inter predicted residual,– Dependent quantization with max QP increased from 51 to 63,– Transform coefficient coding with sign data hiding,– Transform skip residual coding;^ Entropy Coding– Arithmetic coding engine with adaptive double windows probabilityupdate;^ In loop filter– In-loop reshaping,– Deblocking filter with strong longer filter,– Sample adaptive offset,– Adaptive Loop Filter;^ Screen content coding:– Current picture referencing with reference region restriction;^ 360-degree video coding– Horizontal wrap-around motion compensation;^ High-level syntax and parallel processing– Reference picture management with direct reference picture listsignaling, –Tile groups with rectangular shape tile groups;

[0047] Partitioning in VVC

[0048] In VVC, each picture is divided into coding tree units (CTUs) similar toHEVC. A picture may also be divided into slices, tiles, bricks and sub-pictures. CTU may be split into smaller CUs using quaternary tree structure. Each CU may be divided using quad-tree and nested multi-type tree including ternary and binary split. There are specific rules to infer partitioning in in picture boundaries. The redundant split patterns are disallowed in nested multi-type partitioning.

[0049] Cross-component linear model prediction (CCLM)

[0050] To reduce the cross-component redundancy, a cross-component linearmodel (CCLM) prediction mode is used in the VVC, for which the Chroma samplesare predicted based on the reconstructed luma samples of the same CU by using a linear model as follows:pred^(i, j) = α · rec^′(i, j) + β (3-1)where pred^(i, j) represents the predicted chroma samples in a CU and rec^'(i, j)represents the downsampled reconstructed luma samples of the same CU.

[0051] The CCLM parameters (α and β) are derived with at most fourneighbouring chroma samples and their corresponding down-sampled luma samples. Suppose the current chroma block dimensions are W×H, then W’ and H’ are set as: –W’ = W, H’ = H when LM mode is applied;– W’ =W + H when LM-A mode is applied;– H’ = H + W when LM-L mode is applied;

[0052] The above neighboring positions are denoted as S[ 0, −1 ]…S[ W’ − 1,−1 ] and the left neighboring positions are denoted as S[ −1, 0 ]…S[ −1, H’ − 1 ]. Thenthe four samples are selected as: –S[W’ / 4, −1 ], S[ 3 * W’ / 4, −1 ], S[ −1, H’ / 4 ], S[ −1, 3 * H’ / 4 ] when LMmode is applied and both above and left neighbouring samples are available; –S[ W’ / 8, −1 ], S[ 3 * W’ / 8, −1 ], S[ 5 * W’ / 8, −1 ], S[ 7 * W’ / 8, −1 ] whenLM-A mode is applied or only the above neighbouring samples are available; –S[ −1, H’ / 8 ], S[ −1, 3 * H’ / 8 ], S[ −1, 5 * H’ / 8 ], S[ −1, 7 * H’ / 8 ] whenLM-L mode is applied or only the left neighbouring samples are available;

[0053] The four neighboring luma samples at the selected positions are down-sampled and compared four times to find two smaller values: x0A and x1A, and twolarger values: x0B and x1B. Their corresponding Chroma sample values are denoted asy0A, y1A, y0B and y1B. Then xA, xB, yA and yB are derived as:+1)>>1 (3-2).

[0054] Finally, the linear model parameters ^ and ^ are obtained according tothe following equations:^ =^^^^^^^^^^(3-3)β = ^^ − α · ^^ (3-4)

[0055] FIG. 1A shows an example of the location of the left and above samplesand the sample of the current block involved in the CCLM mode.

[0056] The division operation to calculate parameter α is implemented with alook-up table. To reduce the memory required for storing the table, the diff value(difference between maximum and minimum values) and the parameter α are expressedby an exponential notation. For example, diff is approximated with a 4-bit significant part and an exponent. Consequently, the table for 1 / diff is reduced into 16 elements for 16 values of the significand as follows:DivTable [ ] = { 0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0 } (3-5)

[0057] This would have a benefit of both reducing the complexity of thecalculation as well as the memory size required for storing the needed tables.

[0058] Besides the above template and left template as shown in FIG. 1A canbe used to calculate the linear model coefficients together, they also can be used alternatively in the other 2 LM modes, called LM_A, and LM_L modes.

[0059] In LM_A mode, only the above template is used to calculate the linearmodel coefficients. To get more samples, the above template is extended to (W+H). In LM_L mode, only left template is used to calculate the linear model coefficients. To get more samples, the left template is extended to (H+W). For a non-square block, the above template is extended to W+W, the left template is extended to H+H.

[0060] To match the Chroma sample locations for 4:2:0 video sequences, twotypes of downsampling filter are applied to luma samples to achieve 2 to 1 downsampling ratio in both horizontal and vertical directions. The selection of downsampling filter is specified by a SPS level flag.

[0061] The two downsmapling filters are as follows, which are correspondingto “type-0” and “type-2” content, respectively:Rec^′(^, ^) =7)

[0062] Note that only one luma line (general line buffer in intra prediction) isused to make the down-sampled luma samples when the upper reference line is at the CTU boundary.

[0063] This parameter computation is performed as part of the decoding processand is not just as an encoder search operation. As a result, no syntax is used to convey the α and β values to the decoder.

[0064] FIG. 1B shows a Table 1 of Derivation of Chroma prediction mode fromluma mode when cclm_is enabled.

[0065] For Chroma intra mode coding, a total of 8 intra modes are allowed forChroma intra mode coding. Those modes include five traditional intra modes and threecross-component linear model modes (CCLM, LM_A, and LM_L). Chroma modesignaling and derivation process are shown in Table 1 of FIG.1B. Chroma mode codingdirectly depends on the intra prediction mode of the corresponding luma block. Sinceseparate block partitioning structure for luma and Chroma components is enabled in Islices, one Chroma block may correspond to multiple luma blocks. Therefore, forChroma DM mode, the intra prediction mode of the corresponding luma block coveringthe center position of the current Chroma block is directly inherited.

[0066] FIG. 1C shows a Table 2 Unified binarization table for Chromaprediction mode.

[0067] In Table 2 of FIG. 1C, the first bin indicates whether it is regular (0) orLM modes (1). If it is LM mode, then the next bin indicates whether it is LM_CHROMA (0) or not. If it is not LM_CHROMA, next 1 bin indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first bin of the binarization table for the corresponding intra_chroma_pred_mode can be discarded prior to the entropy coding. Or, in other words, the first bin is inferred to be 0 and hence not coded. This single binarization table is used for both sps_cclm_enabled_flag equal to 0 and 1 cases. The first two bins in Error! Referencesource not found. are context coded with its own context model, and the rest bins arebypass coded.

[0068] In addition, in order to reduce luma-Chroma latency in dual tree, whenthe 64x64 luma coding tree node is partitioned with Not Split (and ISP is not used forthe 64x64 CU) or QT, the Chroma CUs in 32x32 / 32x16 Chroma coding tree node areallowed to use CCLM in a following way:– If the 32x32 chroma node is not split or partitioned QT split, all chroma CUsin the 32x32 node can use CCLM; –If the 32x32 chroma node is partitioned with Horizontal BT, and the 32x16child node does not split or uses Vertical BT split, all chroma CUs in the 32x16 chroma node can use CCLM.

[0069] In all the other luma and Chroma coding tree split conditions, CCLM isnot allowed for Chroma CU.

[0070] FIG. 2A and FIG. 2B show reconstructed neighbouring samplesclassified into two classes using a threshold, which is the average of the luma reconstructed neighbouring samples.

[0071] Multi-model LM (MMLM)

[0072] The CCLM included in VVC is extended by adding three Multi-modelLM (MMLM) modes. In each MMLM mode, the reconstructed neighbouring samples are classified into two classes using a threshold which is the average of the luma reconstructed neighbouring samples. The linear model of each class is derived using the Least-Mean-Square (LMS) method. For the CCLM mode, the LMS method is alsoused to derive the linear model. FIG. 2A and FIG. 2B illustrate two luma-to-Chromamodels obtained for luma (Y) threshold of 17. Each luma-to-Chroma model has its ownlinear model parameters α and β. As can be seen from FIG.2B, each luma-to-Chromamodel corresponds to a spatial segmentation of the content (i.e., they correspond to different objects or textures in the scene).

[0073] FIG. 3 shows locations of the samples used for the derivation of CCCMfilter when six reference lines are used, in accordance with example embodiments of the invention.

[0074] Convolutional cross-component model (CCCM)

[0075] An improved version of cross-component prediction, known as CCCM,uses 2D filter kernel to derive the luma-to-Chroma model. The filter coefficients arederived decoder-side using reconstructed set of input data and chroma samples. For the filter coefficient derivation, co-located reference sample areas (consisting of reconstructed luma and chroma samples) are defined for both luma and chroma asshown in FIG. 3 where the typically used 4:2:0 chroma down-sampling has beenapplied. The reference sample area for a given block can be, for example, six lines above and left as shown in FIG. 3, yet any number of reference lines (that can be realized by both the encoder and decoder) can be used. Generally, reference samples can contain any chroma and luma samples that have been reconstructed by both the encoder and decoder. Once the reference samples are determined the filter coefficientscan be derived, for example, using different types of linear regression tools such as ordinary least-squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operator.

[0076] FIG. 4 shows from left: 3-tap vertical, 3-tap horizontal, 5-tap cross, 25-tap diamond.

[0077] The dimensions of the filter kernel can be for example 1^3 (1D vertical),3^1 (1D horizontal), 3^3, 7^7 or any dimensions, and can be shaped (by selecting onlya subset of all possible kernel locations) as a cross or a diamond (as shown in FIG.4) or as any given shape. When referring to the samples within the filter kernel the following notation is used: north (above), east (right), south (below), west (left) andcenter, as illustrated in FIG. 4 using the letters N, E, S, W, C.

[0078] The overall method of reconstructing chroma samples using convolutionbetween a decoder-side obtained filter kernel and a set of input data is referred to as convolutional cross-component model (CCCM) here. The following steps can be applied to perform a CCCM operation: 1. Define co-located reference areas over the luma and chroma components.2. Down-sample the luma samples to match the chroma grid (optional).3. Scan the luma and chroma samples of the reference area and collectavailable statistics (such as auto-correlation matrix and cross-correlation vector) based on the filter shape. 4. Solve the filter coefficients by minimizing squared-error (or any othermetric) based on the available statistics (such as the auto-correlation matrix and cross-correlation vector). 5. Calculate a predicted chroma block by convolving the down-sampled lumasamples with the filter kernel.

[0079] Let us define the (possibly down-sampled) luma samples as a 2D array^(^, ^) indexed using horizontal ^-coordinate and vertical ^-coordinate. Let us alsodefine the co-located chroma samples as a 2D array ^(^, ^) and the filter kernel (i.e.,coefficients) as 3^3 array ^(^, ^). On a sample level we define the convolution between^ and ^ as,

[0080] When using other data terms, such as the non-linear square-root term,the appended convolution becomes,where ^^ are filter coefficients that reside outside of the 2D filter kernel yet havebeen obtained as a part of the system of linear equations that were used to solve the 2D filter coefficients in Step 4 above. Similarly, we can add the bias term to the convolution with,

[0081] FIG. 5 shows an example of four reference lines neighboring to aprediction block.

[0082] Multiple reference line (MRL) intra prediction

[0083] Multiple reference line (MRL) intra prediction uses more reference linesfor intra prediction. In FIG. 5, an example of 4 reference lines is depicted, where the samples of segments A and F are not fetched from reconstructed neighboring samplesbut padded with the closest samples from Segment B and E, respectively. HEVC intra- picture prediction uses the nearest reference line (i.e., reference line 0). In MRL, 2 additional lines (reference line 1 and reference line 3) are used.

[0084] The index of selected reference line (mrl_idx) is signaled and used togenerate intra predictor. For reference line idx, which is greater than 0, only include additional reference line modes in MPM list and only signal mpm index without remaining mode. The reference line index is signaled before intra prediction modes, and Planar mode is excluded from intra prediction modes in case a nonzero reference line index is signaled.

[0085] MRL is disabled for the first line of blocks inside a CTU to prevent usingextended reference samples outside the current CTU line. Also, PDPC is disabled when additional line is used. For MRL mode, the derivation of DC value in DC intra prediction mode for non-zero reference line indices are aligned with that of reference line index 0. MRL requires the storage of 3 neighbouring luma reference lines with a CTU to generate predictions. The Cross-Component Linear Model (CCLM) tool also requires 3 neighbouring luma reference lines for its down-sampling filters. The definition of MLR to use the same 3 lines is aligned as CCLM to reduce the storage requirements for decoders.

[0086] Intra sub-partitions (ISP)

[0087] The intra sub-partitions (ISP) divides luma intra-predicted blocksvertically or horizontally into 2 or 4 sub-partitions depending on the block size. For example, minimum block size for ISP is 4x8 (or 8x4). If block size is greater than 4x8 (or 8x4) then the corresponding block is divided by 4 sub-partitions. It has been notedthat the ^ × 128 (with ^ ≤ 64) and 128 × ^ (with ^ ≤ 64) ISP blocks couldgenerate a potential issue with the 64 × 64 VDPU. For example, an ^ × 128 CU inthe single tree case has an ^ × 128 luma TB and two corresponding^ ^× 64 chromaTBs. If the CU uses ISP, then the luma TB will be divided into four ^ × 32 TBs (onlythe horizontal split is possible), each of them smaller than a 64 × 64 block. However,in the current design of ISP chroma blocks are not divided. Therefore, both chromacomponents will have a size greater than a 32 × 32 block. Analogously, a similarsituation could be created with a 128 × ^ CU using ISP. Hence, these two cases are anissue for the 64 × 64 decoder pipeline. For this reason, the CU sizes that can use ISPis restricted to a maximum of 64 × 64. All sub-partitions fulfil the condition of havingat least 16 samples.

[0088] FIG. 6 shows a Matrix weighted intra prediction process.

[0089] Matrix weighted Intra Prediction (MIP)

[0090] Matrix weighted intra prediction (MIP) method is a newly added intraprediction technique into VVC. For predicting the samples of a rectangular block ofwidth ^ and height ^, matrix weighted intra prediction (MIP) takes one line of Hreconstructed neighbouring boundary samples left of the block and one line of ^ reconstructed neighbouring boundary samples above the block as input. If the reconstructed samples are unavailable, they are generated as it is done in the conventional intra prediction. The generation of the prediction signal is based on the following three steps, which are averaging, matrix vector multiplication and linearinterpolation as shown in FIG. 6.

[0091] Inter prediction in VVC

[0092] Merge list may include the following candidate1) Spatial MVP from spatial neighbour CUs2) Temporal MVP from collocated CUs3) History-based MVP from a FIFO table4) Pairwise average MVP (using the candidates already in the list)5) Zero MVs.

[0093] Merged mode width motion vector difference (MMVD) is to signalMVDs and a resolution index after signaling merge candidate.

[0094] In Symmetric MVD, motion information of list-1 are derived frommotion information of list-0 in bi-prediction case.

[0095] In Affine prediction, several motion vectors are indicated / signaled fordifferent corners of a block, which are used to derive the motion vectors of sub-block. In affine merge, affine motion information of a block is generated based on the normal or affine motion information of the neighboring blocks.

[0096] In Sub-block-based temporal motion vector prediction, motion vectorsof sub-blocks of the current block are predicted from a proper subblocks in the reference frame which are indicated by the motion vector of a spatial neighboring block (if available).

[0097] In Adaptive motion vector resolution (AMVR), precision of MVD issignaled for each CU.

[0098] In Bi-prediction with CU-level weight, an index indicated the weightvalues for weighted average of two prediction block.

[0099] Bi-directional optical flow (BDOF) refines the motion vectors in bi-prediction case. BDOF generates two prediction blocks using the signaled motion vectors. Then a motion refinement is calculated two minimize the error between two prediction blocks using their gradient values. The final prediction blocks are refined using the motion refinement and gradient values.

[0100] Bi-prediction with CU-level weight (BCW) and weighted prediction(WP)

[0101] In HEVC, the bi-prediction signal is generated by averaging twoprediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bi-prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals.

[0102] Five weights are allowed in the weighted averaging bi-prediction, ^ ∈{−2, 3, 4, 5, 10}. For each bi-predicted CU, the weight w is determined in one of twoways: 1) for a non-merge CU, the weight index is signalled after the motion vector difference; 2) for a merge CU, the weight index is inferred from neighbouring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width times CU height is greater than or equal to 256). For low- delay pictures, all 5 weights are used. For non-low-delay pictures, only 3 weights (w∈{3,4,5}) are used.

[0103] At the encoder, fast search algorithms are applied to find the weightindex without significantly increasing the encoder complexity. These algorithms are summarized as follows. For further details readers are referred to the VTM software and document JVET-L0646. When combined with AMVR, unequal weights are only conditionally checked for 1-pel and 4-pel motion vector precisions if the current picture is a low-delay picture.

[0104] When combined with affine, affine ME will be performed for unequalweights if and only if the affine mode is selected as the current best mode.

[0105] When the two reference pictures in bi-prediction are the same, unequalweights are only conditionally checked.

[0106] Unequal weights are not searched when certain conditions are met,depending on the POC distance between current picture and its reference pictures, the coding QP, and the temporal level.

[0107] The BCW weight index is coded using one context coded bin followedby bypass coded bins. The first context coded bin indicates if equal weight is used; andif unequal weight is used, additional bins are signalled using bypass coding to indicate which unequal weight is used.

[0108] Weighted prediction (WP) is a coding tool supported by the H.264 / AVCand HEVC standards to efficiently code video content with fading. Support for WP was also added into the VVC standard. WP allows weighting parameters (weight and offset) to be signalled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weight(s) and offset(s) of the corresponding reference picture(s) are applied. WP and BCW are designed for different types of video content. In order to avoid interactions between WP and BCW, which will complicate VVC decoder design, if a CU uses WP, then the BCW weight index is not signalled, and w is inferred to be 4 (i.e. equal weight is applied). For a merge CU, the weight index is inferred from neighbouring blocks based on the merge candidate index. Thiscan be applied to both normal merge mode and inherited affine merge mode. Forconstructed affine merge mode, the affine motion information is constructed based onthe motion information of up to 3 blocks. The BCW index for a CU using theconstructed affine merge mode is simply set equal to the BCW index of the first control point MV.

[0109] In VVC, CIIP and BCW cannot be jointly applied for a CU. When a CUis coded with CIIP mode, the BCW index of the current CU is set to 2, e.g. equal weight.

[0110] Combined inter and intra prediction (CIIP)

[0111] In VVC, when a CU is coded in merge mode, if the CU contains at least64 luma samples (that is, CU width times CU height is equal to or larger than 64), and if both CU width and CU height are less than 128 luma samples, an additional flag is signaled to indicate if the combined inter / intra prediction (CIIP) mode is applied to the current CU. As its name indicates, the CIIP prediction combines an inter prediction signal with an intra prediction signal. The inter prediction signal in the CIIP mode ^^^^^^is derived using the same inter prediction process applied to regular merge mode; and the intra prediction signal ^^^^^^is derived following the regular intra prediction process with the planar mode. Then, the intra and inter prediction signals are combinedusing weighted averaging, where the weight value is calculated depending on the coding modes of the top and left neighbouring blocks (depicted in Error! Reference source not found. 7) as follows: –If the top neighbor is available and intra coded, then set isIntraTop to 1,otherwise set isIntraTop to 0; –If the left neighbor is available and intra coded, then set isIntraLeft to 1,otherwise set isIntraLeft to 0; –If (isIntraLeft + isIntraTop) is equal to 2, then wt is set to 3;– Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, then wt is set to 2;– Otherwise, set wt to 1.

[0112] FIG. 7 shows Top and left neighboring blocks used in CIIP weightderivation.

[0113] The CIIP prediction is formed as shown in FIG. 7.

[0114] Local illumination compensation (LIC)

[0115] LIC is an inter prediction technique to model local illumination variationbetween current block and its prediction block as a function of that between current block template and reference block template. The parameters of the function can be denoted by a scale α and an offset β, which forms a linear equation, that is, α*p[x]+β to compensate illumination changes, where p[x] is a reference sample pointed to by MV at a location x on reference picture. Since α and β can be derived based on current block template and reference block template, no signaling overhead is required for them, except that an LIC flag is signaled for AMVP mode to indicate the use of LIC.

[0116] The local illumination compensation proposed in JVET-O0066 is usedin ECM for uni-prediction inter CUs with the following modifications: • Intra neighbor samples can be used in LIC parameter derivation; • LIC is disabled for blocks with less than 32 luma samples; • For both non-subblock and affine modes, LIC parameter derivation is performed based on the template block samples corresponding to the current CU, instead of partial template block samples corresponding to first top-left 16x16 unit; • Samples of the reference block template are generated by using MC with the block MV without rounding it to integer-pel precision.

[0117] Handling of out-of-boundary samples (OOB)

[0118] In bi-directional motion compensation the out of boundary (OOB)prediction samples are discarded and only the non-OOB predictors, when available, areused to generate the final predictor. Specifically, let ^^^_^^,^ and ^^^_^^,^ denote theposition of one prediction sample in one current block, ^^_^^^,^^and ^^_^^^,^^(x = 0,1) denote the MV of the current block; ^^^^^^^^^^^, ^^^^^^^^^^^^, ^^^^^^^^^^and ^^^^^^^^^^^^^are the positions of four boundaries of the picture.

[0119] One prediction sample is regarded as OOB when at least one of thefollowing conditions is satisfied:(^^^_^^,^ + ^^_^^^,^^) > (^^^^^^^^^^^^^+ half_pixel),(^^^_^^ + ^^_^ ^^,^ ^,^ ) < (^^^^^^^^^^- half_pixel)where half_pixel is equal to 8 that represents the half-pel sample distance in the1 / 16-pel sample precision.

[0120] After examining the OOB condition for each sample, the final predictionsamples of one bi-directional block are generated as follows: If ^^^,^^is OOB and ^^^,^^is non-OOB ^ ^^^^^ ^,^ = ^^^,^^else if ^^^,^^is non-OOB and ^^^,^^is OOB ^ ^^^^^= ^^^ ^,^ ^,^else

[0121] OOB checking process is also applicable when BCW is enabled.

[0122] Finally, note this sample-adaptive bi-prediction process only applies toprediction units for which at least a reference bock is first detected as partially or entirely out-of-bounds. Thus, a block-level OOB criteria is first checked. If both prediction blocks are non-OOB, then the usual bi-prediction takes place.

[0123] In-loop filters

[0124] There are totally three in-loop filters in VVC. Besides deblocking filterand SAO (the two loop filters in HEVC), adaptive loop filter (ALF) are applied. The ALF comprises of luma ALF, chroma ALF and cross-component ALF (CC-ALF). The ALF filtering process is designed so that luma ALF, chroma ALF and CC-ALF can be executed in parallel. The order of the filtering process in the VVC is the deblocking filter, SAO and ALF. The SAO in VVC is the same as that in HEVC.

[0125] In VVC, an Adaptive Loop Filter (ALF) with block-based filter adaptionis applied. For the luma component, one among 25 filters is selected for each 4×4 block, based on the direction and activity of local gradients.

[0126] FIG. 8 shows ALF filter shapes (Chroma: 5×5 diamond, luma: 7×7diamond).

[0127] Two diamond filter shapes (as shown in FIG. 8) are used. The 7×7diamond shape is applied for luma component and the 5×5 diamond shape is applied for chroma components.

[0128] For luma component, each 4 × 4 block is categorized into one out of 25classes. The classification index C is derived based on its directionality ^ and aquantized value of activity ^^, as follows: ^= 5^ + ^^

[0129] To calculate ^ and ^^, gradients of the horizontal, vertical and twodiagonal direction are first calculated using 1-D Laplacian:

[0130] Where at least in the above calculation indices ^ and ^ refer to thecoordinates of the upper left sample within the 4 × 4 block and ^(^, ^) indicates areconstructed sample at coordinate (^, ^).

[0131] FIG. 9 shows Subsampled Laplacian calculation.

[0132] To reduce the complexity of block classification, the subsampled 1-DLaplacian calculation is applied. As shown in FIG. 9, the same subsampled positionsare used for gradient calculation of all directions.

[0133] Then ^ maximum and minimum values of the gradients of horizontaland vertical directions are set as:

[0134] The maximum and minimum values of the gradient of two diagonaldirections are set as: ^^^ ^^^^^^,^^ = ^^^(^^^, ^^^), ^^^,^^ = ^^^(^^^, ^^^)

[0135] To derive the value of the directionality ^, these values are comparedagainst each other and with two thresholds and ^^Step 1. If both ^ ^^^^ ≤ ^^ ∙ ^^^^and ^^^^ ≤ ^ ^^^ ∙ ^ ^,^ ^,^ ^^,^^ ^^,^^ are true, ^ is set to 0;Step 2. If ^^^^⁄ ^^^ > ^^^^⁄ ^^^ , continue from Step 3; otherwise^ ^^,^ ^,^ ^^,^^ ^^,^^continue from Step 4; Step 3. set to 2; otherwise ^ is set toStep 4. set to 4; otherwise ^ is set to

[0136] The activity value ^ is calculated as:^is further quantized to the range of 0 to 4, inclusively, and the quantized value isdenoted as ^^.

[0137] For chroma components in a picture, no classification method is applied.

[0138] Before filtering each 4×4 luma block, geometric transformations suchas rotation or diagonal and vertical flipping are applied to the filter coefficients ^(^, ^)and to the corresponding filter clipping values ^(^, ^) depending on gradient valuescalculated for that block. This is equivalent to applying these transformations to the samples in the filter support region. The idea is to make different blocks to which ALF is applied more similar by aligning their directionality.

[0139] Three geometric transformations, including diagonal, vertical flip androtation are introduced: Diagonal: ^^(^, ^) = ^(^, ^), ^^(^, ^) = ^(^, ^),Vertical flip: ^^(^, ^) = ^(^, ^ − ^ − 1), ^^(^, ^) = ^(^, ^ − ^ − 1),Rotation: ^^(^, ^) = ^(^ − ^ − 1, ^), ^^(^, ^) = ^(^ − ^ − 1, ^),where ^ is the size of the filter and 0 ≤ ^, ^ ≤ ^ − 1 are coefficients coordinates, suchthat location (0,0) is at the upper left corner and location (^ − 1, ^ − 1) is at the lowerright corner. The transformations are applied to the filter coefficients f (k, l) and to theclipping values ^(^, ^) depending on gradient values calculated for that block. Therelationship between the transformation and the four gradients of the four directions aresummarized in FIG.1D.

[0140] FIG. 1D shows a Table Mapping of the gradient calculated for one blockand the transformations.

[0141] Mappings as shown in FIG. 1E are of the gradient calculated for oneblock and the transformations.

[0142] At decoder side, when ALF is enabled for a CTB, each sample^(^, ^) within the CU is filtered, resulting in sample value ^′(^, ^) as shown below,≫7^where ^(^, ^) denotes the decoded filter coefficients, ^(^, ^) is the clippingfunction and ^(^, ^) denotes the decoded clipping parameters. The variable k and lvaries between −where L denotes the filter length.

[0143] The clipping function ^(^, ^) = min (^, max(−^, ^)) whichcorresponds to the function ^^^^3 (−^, ^, ^). The clipping operation introduces non-linearity to make ALF more efficient by reducing the impact of neighbor sample values that are too different with the current sample value.

[0144] CC-ALF uses luma sample values to refine each chroma component byapplying an adaptive, linear filter to the luma channel and then using the output of this filtering operation for chroma refinement. FIG.10(a) provides a system level diagram of the CC-ALF process with respect to the SAO, luma ALF and chroma ALF processes.

[0145] Filtering in CC-ALF is accomplished by applying a linear, diamondshaped filter (FIG. 10 (b)) to the luma channel. One filter is used for each chromachannel, and the operation is expressed aswhere (^, ^) is chroma component i location being refined (^^ , ^^) is the lumalocation based on (^, ^), ^^ is filter support area in luma component, ^^(^^, ^^)represents the filter coefficients.

[0146] As shown in FIG. 10, the luma filter support is the region collocatedwith the current chroma sample after accounting for the spatial scaling factor between the luma and chroma planes.

[0147] FIG. 10 shows (a) Placement of CC-ALF with respect to other loopfilters (b) Diamond shaped filter.

[0148] In the VVC reference software, CC-ALF filter coefficients are computedby minimizing the mean square error of each chroma channels with respect to the original chroma content. To achieve this, the VTM algorithm uses a coefficient derivation process similar to the one used for luma alf.

[0149] Specifically, a correlation matrix is derived, and the coefficients arecomputed using a Cholesky decomposition solver in an attempt to minimize a mean square error metric. In designing the filters, a maximum of 8 CC-ALF filters can be designed and transmitted per picture. The resulting filters are then indicated for each of the two chroma channels on a CTU basis.

[0150] Adaptive in-loop filters (such as ALF in VVC) perform luma andchroma filtering in parallel fashion to lower the mean square error (MSE) between the reconstruction and the original samples. More specifically, in VVC the ALF first filtersboth luma and chroma in parallel, and subsequently also applies cross-component filtering (CC-ALF) to further improve the chroma, see FIG.10(a).

[0151] The CC-ALF in VVC uses as input the luma samples before ALF. Usingluma samples as an input to CC-ALF are not optimal always and therefore inaccordance with example embodiments of the invention there is proposed to improveto this particular problem.

[0152] FIG. 11 shows Improved CC-ALF pipeline with additional filteredChroma buffers, in accordance with example embodiments of the invention.

[0153] FIG. 12 shows Improved Chroma ALF pipeline with additional filteredChroma buffers.

[0154] It is noted that in accordance with example embodiments of theinvention as shown in FIG. 11 and FIG. 12 the filtering can be using Gausian smoothing, anisotropic nonlinear smoothing, and / or enhancement.

[0155] In accordance with example embodiments of the invention two ideaswhich can be combinedland separately considered: 1) As an additional input to CC-ALF (see FIG. 11) there is proposed inaccordance with example embodiments of the invention to use filtered Chromareconstruction Chroma buffers (.s an additional input to CC-ALF. Newadditional filter coefficients are trained by encoder and are signaled to the bit stream. From the decoder side signaled coefficients are read and CC-ALF applied; and / or2) As an additional input to Chroma ALF (see FIG. 12) in accordance withexample embodiments of the invention there is proposed to use filtered Chromareconstruction Chroma buffers as an additional input to Chroma ALF. Newadditional filter coefficients are trained by encoder and are signaled to the bit stream. From the decoder side signaled coefficients are read and Chroma ALFapplied.

[0156] Before describing the example embodiments as disclosed herein indetail, reference is made to FIG.13 for illustrating a simplified block diagram of various electronic devices that are suitable for use in practicing the example embodiments of this invention.

[0157] FIG. 13 shows a block diagram of one possible and non-limitingexemplary system in which the example embodiments may be practiced. In FIG.13, a user equipment (UE) 10 is in wireless communication with a wireless network 1 ornetwork, 1 as in FIG. 13. The wireless network 1 or network 1 as in FIG. 13 cancomprise a communication network such as a mobile network e.g., the mobile network 1 or first mobile network as disclosed herein. Any reference herein to a wireless network 1 as in FIG.13 can be seen as a reference to any wireless network as disclosed herein. Further, the wireless network 1 as in FIG. 13 can also comprises hardwired features as may be required by a communication network. A UE is a wireless, typically mobile device that can access a wireless network. The UE, for example, may be a mobile phone (or called a "cellular" phone) and / or a computer with a mobile terminal function. For example, the UE or mobile terminal may also be a portable, pocket,handheld, computer-embedded or vehicle-mounted mobile device and performs alanguage signaling and / or data exchange with the RAN.

[0158] The UE 10 includes one or more processors DP 10A, one or morememories MEM 10B, and one or more transceivers TRANS 10D interconnected through one or more buses. Each of the one or more transceivers TRANS 10D includes a receiver and a transmitter. The one or more buses may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. The one or more transceivers TRANS 10D which can beoptionally connected to one or more antennas for communication to NN 12 (networknode 12) and NN 13 (network Node 13), respectively. The one or more memories MEM 10B include computer program code PROG 10C. The UE 10 communicates withNN 12 and / or NN 13 via a wireless link 11 or 16.

[0159] The NN 12 (NR / 5G / 6G Node B, an evolved NB, or LTE device) is anetwork node such as a master or secondary node base station (e.g., for NR or LTE longterm evolution) that communicates with devices such as NN 13 and UE 10 of FIG. 13.The NN 12 provides access to wireless devices such as the UE 10 to the wirelessnetwork 1. The NN 12 includes one or more processors DP 12A, one or more memoriesMEM 12B, and one or more transceivers TRANS 12D interconnected through one or more buses. In accordance with the example embodiments these TRANS 12D can include X2 and / or Xn interfaces for use to perform the example embodiments. Each of the one or more transceivers TRANS 12D includes a receiver and a transmitter. The one or more transceivers TRANS 12D can be optionally connected to one or more antennas for communication over at least link 11 with the UE 10. The one or more memories MEM 12B and the computer program code PROG 12C are configured tocause, with the one or more processors DP 12A, the NN 12 to perform one or more ofthe operations as described herein. The NN 12 may communicate with another gNB oreNB, or a device such as the NN 13 such as via link 16 or link 18. Further, the link 11,link 16 and / or any other link may be wired or wireless or both and may implement, e.g., an X2 or Xn interface. Further the link 11 and / or link 16 and / or link 18 may be through other network devices such as, but not limited to an NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 device as in FIG. 13. The NN 12 may perform functionalities of an MME (Mobility Management Entity) or SGW (Serving Gateway), such as a User Plane Functionality, and / or an Access Management functionality for LTE and similar functionality for 5G or 6G.

[0160] The NN 13 can be for WiFi or Bluetooth or other wireless deviceassociated with a mobility function device such as an AMF or SMF, further the NN 13 may comprise a NR / 5G / 6G Node B or possibly an evolved NB a base station such as a master or secondary node base station (e.g., for NR or LTE long term evolution) thatcommunicates with devices such as the NN 12 and / or UE 10 and / or the wirelessnetwork 1. The NN 13 includes one or more processors DP 13A, one or more memoriesMEM 13B, one or more network interfaces, and one or more transceivers TRANS 13D interconnected through one or more buses. In accordance with the exampleembodiments these network interfaces of NN 13 can include X2 and / or Xn interfacesfor use to perform the example embodiments. Each of the one or more transceiversTRANS 13D includes a receiver and a transmitter that can optionally be connected to one or more antennas. The one or more memories MEM 13B include computer program code PROG 13C. For instance, the one or more memories MEM 13B and the computer program code PROG 13C are configured to cause, with the one or moreprocessors DP 13A, the NN 13 to perform one or more of the operations as describedherein. The NN 13 may communicate with another mobility function device and / oreNB such as the NN 12 and the UE 10 or any other device using, e.g., link 11 or link16 or link 18 or another link. The link 16 or link 18 as shown in FIG.13 can be used for communication with the NN 12. These links maybe wired or wireless or both and may implement, e.g., an X2 or Xn interface. Further, as stated above the link 11 and / or link 16 and / or link 18 may be through other network devices such as, but not limited to an NCE / MME / SGW device such as the NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 of FIG.13.

[0161] The one or more buses of the device of FIG. 13 may be address, data, orcontrol buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, wireless channels, and the like. For example, the one or more transceivers TRANS 12D, TRANS 13D and / or TRANS 10D may be implemented as a remote radiohead (RRH), with the other elements of the NN 12 being physically in a differentlocation from the RRH, and these devices can include one or more buses that could beimplemented in part as fiber optic cable to connect the other elements of the NN 12 toa RRH.

[0162] It is noted that although FIG. 13 shows a network nodes such as NN 12and NN 13, any of these nodes may can incorporate or be incorporated into an eNodeB or eNB or gNB such as for LTE and NR, and would still be configurable to perform example embodiments.

[0163] Also it is noted that description herein indicates that “cells” performfunctions, but it should be clear that the gNB that forms the cell and / or a user equipment and / or mobility management function device that will perform the functions. In addition, the cell makes up part of a gNB, and there can be multiple cells per gNB.

[0164] The wireless network 1 or any network it can represent may or may notinclude a NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 that may include (NCE) network control element functionality, MME (Mobility Management Entity) / SGW (Serving Gateway) functionality, and / or serving gateway (SGW), and / or MME (Mobility Management Entity) and / or SGW (Serving Gateway) functionality, and / or user data management functionality (UDM), and / or PCF (Policy Control) functionality, and / or Access and Mobility Management Function (AMF) functionality, and / or Session Management (SMF) functionality, and / or Location Management Function (LMF), and / or Authentication Server (AUSF) functionality and which provides connectivity with a further network, such as a telephone network and / or a data communications network (e.g., the Internet), and which is configured to perform any 5G, 6G, and / or NR operations in addition to or instead of other standard operations at the time of this application. The NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 is configurable to perform operations in accordance with example embodiments in any of an LTE, NR, 5G, 6G, and / or any standards based communication technologies being performed or discussed at the time of this application. In addition, it is noted that the operations inaccordance with example embodiments, as performed by the NN 12 and / or NN 13, mayalso be performed at the NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14.

[0165] The NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 includes one ormore processors DP 14A, one or more memories MEM 14B, and one or more network interfaces (N / W I / F(s)), interconnected through one or more buses coupled with the link 13 and / or link 16 and / or link 18. In accordance with the example embodiments these network interfaces can include X2 and / or Xn interfaces for use to perform the example embodiments. The one or more memories MEM 14B include computer program code PROG 14C. The one or more memories MEM14B and the computer program code PROG 14C are configured to, with the one or more processors DP 14A, cause the NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 to perform one or more operations which may be needed to support the operations in accordance with the example embodiments.

[0166] It is noted that that the NN 12 and / or NN 13 and / or UE 10 can beconfigured (e.g. based on standards implementations etc.) to perform functionality of a Location Management Function (LMF). The LMF functionality may be embodied in any of these network devices or other devices associated with these devices. In addition, an LMF such as the LMF of the MME / SGW / UDM / PCF / AMF / SMF / LMF 14 of FIG. 13, as at least described below, can be co-located with UE 10 such as to be separatefrom the NN 12 and / or NN 13 of FIG. 13 for performing operations in accordance withexample embodiments as disclosed herein.

[0167] The wireless Network 1 may implement network virtualization, whichis the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to software containers on a single system. Note that the virtualized entities that result from the network virtualization are still implemented, at some level, using hardware such as processors DP10, DP12A, DP13A, and / or DP14A and memories MEM 10B, MEM 12B, MEM 13B, and / or MEM 14B, and also such virtualized entities create technical effects.

[0168] The computer readable memories MEM 12B, MEM 13B, and MEM14B may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The computer readable memories MEM 12B, MEM 13B, and MEM 14B may be means for performing storage functions. The processors DP10, DP12A, DP13A, and DP14A may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi-core processor architecture, as non-limiting examples. The processors DP10, DP12A, DP13A, and DP14A may bemeans for performing functions, such as controlling the UE 10, NN 12, NN 13, and other functions as described herein.

[0169] In general, various embodiments of any of these devices can include, butare not limited to, cellular telephones such as smart phones, tablets, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback appliances having wireless communication capabilities, Internet appliances permitting wireless Internet access and browsing, tablets with wireless communication capabilities, as well as portable units or terminals that incorporate combinations of such functions.

[0170] Further, the various embodiments of any of these devices can be usedwith a UE vehicle, a High Altitude Platform Station, or any other such type node associated with a terrestrial network or any drone type radio or a radio in aircraft or other airborne vehicle or a vessel that travels on water such as a boat.

[0171] As similarly stated above, in accordance with example embodiments ofthe invention two ideas which can be combinedly and separately considered. 1) As an additional input to CC-ALF (see FIG. 11) in accordance with exampleembodiments of the invention there is proposed to use smoothed Chroma reconstruction Chroma buffers as an additional input to CC-ALF. Newadditional filter coefficients are trained by encoder and are signaled to the bit stream. From the decoder side signaled coefficients are read and CC-ALF applied; 2) As an additional input to Chroma ALF (see FIG. 12) in accordance withexample embodiments of the invention there is proposed to use smoothed Chroma reconstruction Chroma buffers as an additional input to Chroma ALF.New additional filter coefficients are trained by encoder and are signaled to the bit stream. From the decoder side signaled coefficients are read and Chroma ALF applied.

[0172] For smoothing input, in accordance with example embodiments of theinvention there is proposed to choose before deblocking filter Chroma reconstructionbuffers. This part, smoothing buffer selection can be changed to different stages, i.e.,after deblocking filter Chroma reconstruction buffers or after SAO Chromareconstruction buffers.

[0173] In VVC, ALF uses Wiener filters to lower the MSE of luma and chromareconstructions. The filter coefficients are either selected from a fixed set of coefficients using a signalled index or explicitly signalled from the encoder to the decoder. In both cases signalling is involved. A cross-component variant called CC-ALF improves chroma by applying a Wiener filter to the luma component in order to obtain a correction term for the chroma component, see FIG. 10. In all ALF related filter derivations, the goal is (at the encoder) to minimize the error against the original luma and chroma.

[0174] The ALF pipeline in VVC is summarized in the following steps,1a. Apply luma ALF on the reconstructed luma,2a. Apply chroma ALF on the reconstructed chroma,3a. Apply CC-ALF on reconstructed luma (i.e., the input of Step 1a) toobtain a chroma correction, 4a. Add the output of Step 3a to the output of Step 2a.

[0175] In this report in accordance with example embodiments of the inventionthere is proposed an improved CC-ALF pipeline as shown in FIG. 11. An additionalfiltering, i.e., smoothing stage, filtered, i.e., smoothed chroma buffers are used as anadditional input to CC-ALF.

[0176] The following steps describe the improved pipeline of CC-ALF,1b. Apply luma ALF on the reconstructed luma,2b. Perform filtering, i.e., smoothing operation on chroma buffers andderive a filtered, i.e., smoothed version of chroma buffers,3b. Apply chroma ALF,4b. Apply CC-ALF to obtain a chroma correction with an additional inputobtained from Step 2b (see FIG. 11),5b. Combine the output of Step 4b to the output of Step 3b.

[0177] In accordance with example embodiments of the invention there isproposed an improved chroma ALF pipeline as shown in FIG. 12. An additionalfiltering, i.e., smoothing stage, filtered, i.e., smoothed chroma buffers are used as anadditional input to chroma ALF.

[0178] The following steps describe the improved pipeline of chroma ALF:1c. Apply luma ALF on the reconstructed luma;2c. Perform filtering, i.e., smoothing operation on chroma buffers andderive a filtered, i.e., smoothed version of chroma buffers;3c. Apply chroma ALF with an additional input obtained from Step 2c (seeFIG. 12);4c. Apply CC-ALF to obtain a chroma correction;5c. Combine the output of Step 4c to the output of Step 3c.

[0179] The Step 2b and 2c can be applied before Step 1b and 1c respectivelydepending on software design.

[0180] Filtering can be performed data point by point. For each point, wecalculate the kernel function values and calculate the weighted average of data points weighted by kernel function values.

[0181] The filtering can be applied by Gaussian smoothing which is isotropicsmoothing. In a test implementation in accordance with example embodiments of the invention can be using applied Gaussian smoothing.

[0182] Gaussian functions are widely used in statistics to describe the normaldistributions, in signal processing to define Gaussian filters. Gaussian functions are used in image processing where two-dimensional Gaussians are used for Gaussian blurs, and in mathematics to solve heat equations and diffusion equations.

[0183]

[0184] Encoder side Chroma ALF and CC-ALF statistics derivation andcoefficient training is kept same as in VVC. We only increase the number of coefficients by an additional filter shape (e.g., 3x3 diamond shape) which corresponds to an additional input buffer. Encoder signaling is kept the same.

[0185] Decoder side Chroma ALF and CC-ALF filtering and signaling are keptthe same as in VVC. Only our additional coefficients are read from the bit stream andused while applying filtering.

[0186] FIG. 14 shows a method in accordance with example embodiments ofthe invention which may be performed by an apparatus.

[0187] FIG. 14 illustrates operations which may be performed by a device suchas, but not limited to, a device such as network node (e.g., the UE 10 as in FIG. 13). Asshown in block 1410 of FIG.14 there is applying a cross component adaptive loop filter on a reconstructed luma to obtain a reconstructed Chroma. As shown in block 1420 ofFIG. 14 there is applying a Chroma adaptive loop filter on the reconstructed Chroma toobtain a Chroma correction. Then as shown in block 1430 of FIG. 14 there is using theChroma correction as an additional input to improve a cross-component adaptive loopfilter for a video of or image stream.

[0188] In accordance with the example embodiments as described in theparagraph above, wherein the performance monitoring metric based on calculation function comprises a machine learning model.

[0189] In accordance with the example embodiments as described in theparagraphs above, wherein the Chroma correction is used as as an additional input tothe cross component adaptive loop filter to derive filtered chroma reconstructionbuffers.

[0190] In accordance with the example embodiments as described in theparagraphs above, wherein the filtered Chroma reconstruction buffers are used for theadditional input to the cross component adaptive loop filter.

[0191] In accordance with the example embodiments as described in theparagraphs above, wherein the cross-component adaptive loop filter is applying a Wiener filter to a luma component to correct at least one of a chroma component or the luma component.

[0192] In accordance with the example embodiments as described in theparagraphs above, wherein the cross-component adaptive loop filter comprises at leastone of an adaptive loop filter luma or an adaptive loop filter Chroma.

[0193] In accordance with the example embodiments as described in theparagraphs above, wherein the additional input is using the adaptive loop filter Chroma.

[0194] In accordance with the example embodiments as described in theparagraphs above, wherein an additional filter shape is used to increase at least one ofa number of filter coefficients or clipping values trained as an additional input buffer for the additional input.

[0195] In accordance with the example embodiments as described in theparagraphs above, wherein at least one of the increased filter coefficients or clippingvalues are used on the video or image stream during the additional input to improve a cross-component adaptive loop filter.

[0196] In accordance with the example embodiments as described in theparagraphs above, wherein the filtered Chroma reconstruction Chroma bufferscomprise a filtered Chroma.

[0197] In accordance with the example embodiments as described in theparagraphs above, wherein the filtered Chroma is based on isotropic smoothingcomprising at least one of Gaussian smoothing, anisotropic nonlinear smoothing, orenhancing.

[0198] In accordance with the example embodiments as described in theparagraphs above, wherein the cross-component adaptive loop filter comprises lumafiltering and Chroma filtering.

[0199] A non-transitory computer-readable medium (MEM 10B as in FIG. 7)storing program code (PROG 10C as in FIG.13), the program code executed by at least one processor (DP 10A as in FIG.13) to perform the operations as at least described in the paragraphs above.

[0200] In accordance with an example embodiment as described above there isan apparatus comprising: means for applying (one or more transceivers 10D; MEM 10B; PROG 10C; and DP 10A as in FIG.13) a cross component adaptive loop filter on a reconstructed luma to obtain a reconstructed Chroma; means for applying (one or more transceivers 10D; MEM 10B; PROG 10C; and DP 10A as in FIG.13) a Chromaadaptive loop filter on the reconstructed Chroma to obtain a Chroma correction; andmeans for using (one or more transceivers 10D; MEM 10B; PROG 10C; and DP 10Aas in FIG. 13) the Chroma correction as an additional input to improve a cross-component adaptive loop filter for a video or image stream.

[0201] In the example aspects according to the paragraph above, wherein atleast the means for applying and the means for using comprises a non-transitory computer readable medium [MEM 12B and / or MEM 13B] encoded with a computer program [PROG 12C and / or PROG 13C] executable by at least one processor [DP 12A and / or DP 13A].

[0202] It is noted that computer-implemented inventions (CII) may be claimedas apparatus claims, method claims, and software claims. In some jurisdictions, such as in Europe, signal claims can also be made. In the U.S., a software claim must be claimed as a non-transitory computer program product or a non-transitory computer readable medium.

[0203] The term “non-transitory,” as used herein, is a limitation of the mediumitself (i.e., tangible, not a signal ) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0204] In other jurisdictions, a software claim can be claimed as a computerprogram, a data structure, and / or a computer readable medium.

[0205] Further, in accordance with example embodiments of the invention thereis circuitry for performing operations in accordance with example embodiments of the invention as disclosed herein. This circuitry can include any type of circuitry including content coding circuitry, content decoding circuitry, processing circuitry, image generation circuitry, data analysis circuitry, etc.). Further, this circuitry can include discrete circuitry, application-specific integrated circuitry (ASIC), and / or field-programmable gate array circuitry (FPGA), etc. as well as a processor specificallyconfigured by software to perform the respective function, or dual-core processors with software and corresponding digital signal processors, etc.). Additionally, there are provided necessary inputs to and outputs from the circuitry, the function performed by the circuitry and the interconnection (perhaps via the inputs and outputs) of the circuitry with other components that may include other circuitry in order to perform example embodiments of the invention as described herein.

[0206] In accordance with example embodiments of the invention as disclosedin this application this application, the “circuitry” provided can include at least one or more or all of the following:(a) hardware-only circuit implementations (such as implementations in onlyanalog and / or digital circuitry);(b) combinations of hardware circuits and software, such as (as applicable):(i) a combination of analog and / or digital hardware circuit(s) withsoftware / firmware; and(ii) any portions of hardware processor(s) with software (including digital signalprocessor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions, such as functions or operations in accordance with example embodiments of the invention as disclosed herein); and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.”

[0207] In accordance with example embodiments of the invention, there isadequate circuitry for performing at least novel operations in accordance with example embodiments of the invention as disclosed in this application, this `circuitry` as may be used herein refers to at least the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); and (b) to combinations of circuits and software (and / or firmware), such as (as applicable): (i) to a combination of processor(s) or (ii) to portions of processor(s) / software(including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (c) to circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present.

[0208] This definition of `circuitry` applies to all uses of this term in thisapplication, including in any claims. As a further example, as used in this application, the term "circuitry" would also cover an implementation of merely a processor (or multiple processors) or portion of a processor and its (or their) accompanying software and / or firmware. The term "circuitry" would also cover, for example and if applicable to the particular claim element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or other network device.

[0209] In general, the various embodiments may be implemented in hardwareor special purpose circuits, software, logic or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0210] Embodiments of the inventions may be practiced in various componentssuch as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available forconverting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0211] The word "exemplary" is used herein to mean "serving as an example,instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this Detailed Description are exemplary embodiments provided to enable persons skilled in the art to make or use the invention and not to limit the scope of the invention which is defined by the claims.

[0212] The foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of the best method and apparatus presently contemplated by the inventors for carrying out the invention. However,various modifications and adaptations may become apparent to those skilled in therelevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of example embodiments of this invention will still fall within the scope of this invention.

[0213] It should be noted that the terms "connected," "coupled," or any variantthereof, mean any connection or coupling, either direct or indirect, between two or more elements, and may encompass the presence of one or more intermediate elements between two elements that are "connected" or "coupled" together. The coupling or connection between the elements can be physical, logical, or a combination thereof. As employed herein two elements may be considered to be "connected" or "coupled" together by the use of one or more wires, cables and / or printed electrical connections, as well as by the use of electromagnetic energy, such as electromagnetic energy having wavelengths in the radio frequency region, the microwave region and the optical (both visible and invisible) region, as several non-limiting and non-exhaustive examples.

[0214] Furthermore, some of the features of the preferred embodiments of thisinvention could be used to advantage without the corresponding use of other features.As such, the foregoing description should be considered as merely illustrative of the principles of the invention, and not in limitation thereof.

Claims

CLAIMS What is claimed is:

1. An apparatus, comprising:at least one processor; and at least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to: apply a cross component adaptive loop filter on a reconstructed luma to obtain a reconstructed Chroma; apply a Chroma adaptive loop filter on the reconstructed Chroma to obtain aChroma correction; anduse the Chroma correction as an additional input to improve a cross-component adaptive loop filter for a video or image stream.

2. The apparatus of claim 1, wherein the Chroma correction is used as anadditional input to the cross component adaptive loop filter to derive filtered chromareconstruction buffers.

3. The apparatus of claim 2, wherein the filtered chroma reconstruction buffersare used as the additional input to the cross component adaptive loop filter.

4. The apparatus of claim 1, wherein the cross-component adaptive loop filter isapplying a Wiener filter to a luma component to correct at least one of a chromacomponent or the luma component.

5. The apparatus of claim 1, wherein the cross-component adaptive loop filter comprises at least one of an adaptive loop filter luma or an adaptive loop filter Chroma.

6. The apparatus of claim 5, wherein the additional input is using the adaptiveloop filter Chroma.

7. The apparatus of claim 1, wherein an additional filter shape is used to increase at least one of a number of filter coefficients or clipping values trained as an additional input buffer for the additional input.

8. The apparatus of claim 7, wherein at least one of the increased filtercoefficients or clipping values are used on the video or image stream during theadditional input to improve the cross-component adaptive loop filter.

9. The apparatus of claim 1, wherein the filtered Chroma reconstruction Chromabuffers comprise a filtered Chroma10. The apparatus of claim 9, wherein the filtered Chroma is based on an isotropicsmoothing comprising at least one of gaussian smoothing, anisotropic nonlinearsmoothing, or enhancing.

11. The apparatus of claim 1, wherein the cross-component adaptive loop filtercomprises luma filtering and Chroma filtering.

12. A method, comprising: applying a cross component adaptive loop filter on a reconstructed luma to obtain a reconstructed Chroma; applying a Chroma adaptive loop filter on the reconstructed Chroma to obtaina Chroma correction; andusing the Chroma correction as an additional input to improve a cross-component adaptive loop filter for a video or image stream.

13. The method of claim 12, wherein the Chroma correction is used as anadditional input to the cross component adaptive loop filter to derive filtered chromareconstruction buffers.

14. The method of claim 13, wherein the filtered chroma reconstruction buffersare used as the additional input to the cross component adaptive loop filter.

15. The method of claim 12, wherein the cross-component adaptive loop filter isapplying a Wiener filter to a luma component to correct at least one of a chromacomponent or the luma component.

16. The method of claim 12, wherein the cross-component adaptive loop filter comprises at least one of an adaptive loop filter luma or an adaptive loop filter Chroma.

17. The method of claim 16, wherein the additional input is using the adaptiveloop filter Chroma.

18. The method of claim 12, wherein an additional filter shape is used to increase at least one of a number of filter coefficients or clipping values trained as an additional input buffer for the additional input.

19. The method of claim 18, wherein at least one of the increased filtercoefficients or clipping values are used on the video or image stream during the additional input to improve a cross-component adaptive loop filter.

20. The method of claim 12, wherein the filtered Chroma reconstruction Chromabuffers comprise a filtered Chroma21. The method of claim 20, wherein the filtered Chroma is based on isotropicsmoothing comprising at least one of Gaussian smoothing, anisotropic nonlinearsmoothing, or enhancing.

22. The method of claim 12, wherein the cross-component adaptive loop filtercomprises luma filtering and Chroma filtering.

Images