Methods and devices for prediction dependent residual scaling for video coding

Abstract

Methods and devices for reducing decoding latency introduced by LMCS are provided. In one method, during decoding of a coding unit (CU), a plurality of reconstructed luminance samples are selected from a first predetermined region, the first predetermined region is adjacent to a second predetermined region where the CU is located, the average of the plurality of reconstructed luminance samples is calculated, and when deriving a chroma residual scaling factor for decoding the CU, the average of the plurality of reconstructed luminance samples is used directly without any clipping.

Description

Technology Field

[0001] This application is a continuation of PCT application number PCT / US2021 / 024493 filed on March 26, 2021, which is based on and claims priority to provisional application number 63 / 001,229 filed on March 27, 2020, and the disclosures of such provisional application are incorporated herein by reference in their entirety for all purposes.

[0002] The present disclosure generally relates to video coding and compression. More specifically, the present disclosure relates to systems and methods for performing video coding using prediction-dependent residual scaling for code units. Background Technology

[0003] This section provides background information regarding the present disclosure. The information contained within this section is not necessarily to be interpreted as prior art.

[0004] Any of the various video coding techniques may be used to compress video data. Video coding may be performed according to one or more video coding standards. Some exemplary video coding standards include Versatile Video Coding (VVC), Joint Exploration Test Model (JEM) coding, High-Efficiency Video Coding (H.265 / HEVC), Advanced Video Coding (H.264 / AVC), and Moving Picture Experts Group (MPEG) coding.

[0005] Video coding generally utilizes predictive methods (e.g., inter-prediction, intra-prediction, etc.) that take advantage of the redundancy inherent in video images or sequences. One goal of video coding techniques is to compress video data using a lower bit rate while avoiding or minimizing degradations in video quality.

[0006] The first version of the HEVC standard was finalized in October 2013, and it provides approximately 50% bitrate reduction or equivalent perceptual quality compared to the previous generation video coding standard, H.264 / MPEG AVC. While the HEVC standard offers significant coding improvements over its predecessor, there is evidence that superior coding efficiency can be achieved with additional coding tools. Based on this, both VCEG and MPEG have begun exploring new coding technologies for future video coding standardization. To initiate important research into advanced technologies capable of enabling substantial improvements in coding efficiency, a Joint Video Exploration Team (JVET) was formed in October 2015 by ITU-T VECG and ISO / IEC MPEG. A single reference software referred to as the Joint Exploration Model (JEM) was maintained by JVET by integrating several additional coding tools on top of the HEVC test model (HM).

[0007] In October 2017, a Call for Proposals (CfP) regarding video compression capabilities exceeding HEVC was issued by ITU-T and ISO / IEC. In April 2018, 23 CfP responses were received and evaluated at the 10th JVET meeting, demonstrating a compression efficiency gain approximately 40% better than HEVC. Based on these evaluation results, JVET launched a new project to develop a new generation video coding standard named Universal Video Coding (VVC). In the same month, a reference software referred to as the VVC Test Model (VTM) was established to demonstrate a reference implementation of the VVC standard.

[0008] Prediction methods used in video coding typically involve performing spatial prediction (intra-frame prediction) and temporal prediction (inter-frame prediction) to reduce or eliminate redundancy inherent in video data, and are typically associated with block-based video coding. Like HEVC, VVC is built on a block-based hybrid video coding framework.

[0009] In block-based video coding, input video signals are processed block by block. For each block (also known as a Coding Unit, CU), spatial prediction and / or temporal prediction can be performed. In newer video coding standards, such as the current VVC design, blocks can be further partitioned based on a multi-type tree structure that includes not only quad-trees but also binary-trees and / or ternary-trees. This allows for better accommodation of various local characteristics.

[0010] Spatial prediction (also known as "intra-prediction") uses pixels from samples of already coded neighboring blocks (referred to as reference samples) within the same video picture / slice to predict the current block. Spatial prediction reduces the spatial redundancy inherent in the video signal.

[0011] During the decoding process, the video bit-stream is first entropy-decoded in the entropy decoding unit. Coding mode and prediction information are sent to the spatial prediction unit (if intra-coded) or the temporal prediction unit (if inter-coded) to form prediction blocks. Residual transform coefficients are sent to the inverse quantization unit and inverse transform unit to reconstruct the residual block. Then, the prediction block and the residual block are added together. The reconstructed block may also pass through in-loop filtering before being stored in the reference picture store. Subsequently, the reconstructed video in the reference picture store is output to drive the display device and is also used to predict future video blocks.

[0012] In newer video coding standards, such as the current VVC design, a coding tool called Luma Mapping with Chroma Scaling (LMCS) can be applied before in-loop filtering. LMCS aims to adjust the dynamic range of the input signal to improve coding efficiency.

[0013] However, current LMCS designs use different domain mappings in various decoding modules, resulting in computational complexity and additional requirements for on-chip memory. Furthermore, current LMCS designs use different luma prediction sample values ​​to derive the luma scaling factor and chroma scaling factor, which introduces additional complexity. Furthermore, the current design of LMCS increases the latency of chroma residual sample reconstruction because it requires deferring the reconstruction of chroma residual samples until after the successful completion of luminance prediction samples (this also requires the successful completion of the sequential application of complex inter-mode coding tools such as Decoder-side Motion Vector Derivation (DMVR), Bi-Directional Optical Flow (BDOF), and Combined Inter and Intra Prediction (CIIP). The current design of LMCS also performs unnecessary clipping operations during the chroma residual scaling factor derivation process. This further increases computational complexity and the additional requirements for on-chip memory.

[0014] This section provides a general overview of the disclosure and is not an exclusive disclosure of the full scope or all features of the disclosure.

[0015] According to a first embodiment of the present application, a method for video coding is provided. The method comprises selecting a plurality of reconstructed luminance samples from a first predetermined region during decoding of a CU, wherein the first predetermined region is adjacent to a second predetermined region in which the CU is located, calculating the average of the plurality of reconstructed luminance samples, and using the average of the plurality of reconstructed luminance samples directly without any clipping when deriving a chroma residual scaling factor for decoding the CU.

[0016] According to a second embodiment of the present application, a computing device is provided. The computing device comprises one or more processors, a memory, and a plurality of programs stored in the memory. When the programs are executed by one or more processors, they cause the computing device to perform operations as described above in the first embodiment of the present application.

[0017] According to a third embodiment of the present application, a non-transient computer-readable storage medium stores a plurality of programs for execution by a computing device having one or more processors. When executed by one or more processors, the programs cause the computing device to perform operations as described above in the first embodiment of the present application. Brief explanation of the drawing

[0018] Hereinafter, a set of exemplary, non-limiting embodiments of the present disclosure will be described in conjunction with the accompanying drawings. Based on the examples presented herein, variations of structures, methods, or functions may be implemented by those skilled in the art, and all such variations are included within the scope of the present disclosure. Where there is no conflict, the teachings of different embodiments may be combined with one another, but are not required to do so. Figure 1 is a block diagram presenting an exemplary block-based hybrid video encoder that can be used in conjunction with many video coding standards. Figure 2 is a block diagram presenting an exemplary video decoder that can be used in conjunction with many video coding standards. Figure 3 is an example of block partitions in a multi-type tree structure that can be used in conjunction with many video coding standards. Figure 4 is a flowchart illustrating a decoding process to which LMCS is applied. Figure 5 is an example of a BDOF process. Figure 6 is a flowchart illustrating the workflow of chroma residual scaling in LMCS when DMVR, BDOF, and CIIP are all enabled. Figure 7 is a flowchart illustrating the steps of the Prediction Dependent Residual Scaling (PDRS) procedure. Figure 8 is a flowchart illustrating the workflow of the decoding process when the PDRS procedure is applied in the LMCS process. Figure 9 is an example of a residual mapping error resulting from using only predicted samples to derive a scaling factor. Figure 10 is a flowchart illustrating the steps of a single chroma sample reconstruction procedure. FIG. 11 is a flowchart illustrating the steps of the second chroma sample reconstruction procedure. FIG. 12 is a flowchart illustrating the workflow of the LMCS decoding process in one example of a second chroma sample reconstruction procedure, where DMVR, BDOF, and CIIP are not applied to generate luminance prediction samples for chroma scaling. FIG. 13 is a flowchart illustrating the workflow of the LMCS decoding process in the second example of the second chroma sample reconstruction procedure, where an initial uni-prediction signal is applied to generate luminance prediction samples for chroma scaling. Figure 14 is a flowchart illustrating the steps of a chroma residual sample reconstruction procedure. FIG. 15 is a flowchart illustrating the workflow of the LMCS decoding process in one or more embodiments of the chroma residual sample reconstruction procedure. Figure 16 is a flowchart illustrating the workflow of the LMCS decoding process in one different embodiment of the chroma residual sample reconstruction procedure. FIG. 17 is a flowchart illustrating the steps of the second chroma residual sample reconstruction procedure. FIG. 18 is a flowchart illustrating the steps of a first embodiment of the present disclosure. FIG. 19 is an example of regions related to an embodiment of the first embodiment of the present disclosure. Specific details for implementing the invention

[0019] Reference to exemplary embodiments will now be made in detail, and such examples are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, wherein identical numbers in different drawings represent identical or similar elements unless otherwise indicated. The embodiments presented in the following description of examples do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with embodiments related to the present disclosure as described in the appended claims.

[0020] The terms used in this disclosure are intended to illustrate specific examples rather than to limit this disclosure. Singular forms as used in this disclosure as well as in the appended claims also refer to plural forms unless the context clearly implies otherwise. It should be understood that the term "and / or" as used herein refers to any one or more of the listed related items or any possible combination thereof.

[0021] Although terms such as “first,” “second,” “third,” etc. may be used in this specification to describe various information, it will be understood that such information should not be limited by these terms. These terms are used merely to distinguish one category of information from another. For example, without departing from the scope of this disclosure, the first information may be named the second information, and similarly, the second information may also be named the first information. As used in this specification, the term “if ~” may be understood to mean “when ~” or “in the case of ~” or “in response to ~” depending on the context.

[0022] Throughout this specification, references in the singular or plural to “one embodiment,” “an example,” “another example,” etc., mean that one or more specific features, structures, or characteristics described in connection with the example are included in at least one embodiment of this disclosure. Accordingly, the appearance of phrases in the singular or plural throughout this specification, such as “in one embodiment,” “in an example,” “in another example,” etc., does not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable manner.

[0023] In order to clearly explain the technical solutions of the embodiments of the present disclosure and to facilitate understanding for those skilled in the art of the present invention, a number of implementation modes are listed in the embodiments of the present disclosure. It is obvious that those skilled in the art of the present invention will understand that the various embodiments provided in the embodiments of the present disclosure may be practiced independently, or may be combined with methods of other embodiments in the embodiments of the present disclosure for practice, or may be practiced independently or after being combined with some methods in other related technologies. No limitations are made in the embodiments of the present disclosure.

[0024] Conceptually, many video coding standards, including those previously mentioned in the background section, are similar. For example, virtually all video coding standards use block-based processing and share similar video coding block diagrams to achieve video compression.

[0025] FIG. 1 shows a block diagram of an exemplary block-based hybrid video encoder (100) that can be used in conjunction with many video coding standards. In the encoder (100), a video frame is partitioned into multiple video blocks for processing. For each given video block, a prediction is formed based on an inter-prediction approach or an intra-prediction approach. In inter-prediction, one or more predictors are formed through motion estimation and motion compensation based on pixels from previous reconstructed frames. In intra-prediction, predictors are formed based on reconstructed pixels in the current frame. Through mode determination, the best predictor can be selected to predict the current block.

[0026] A prediction residual representing the difference between the current video block and its predictor is sent to a transform circuitry (102). Then, the transform coefficients are sent from the transform circuitry (102) to a quantization circuitry (104) for entropy reduction. Then, the quantized coefficients are supplied to an entropy coding circuitry (106) to generate a compressed video bitstream. As shown in FIG. 1, prediction-related information (110) from the inter-prediction circuitry and / or intra-prediction circuitry (112), such as video block partition info, motion vectors, reference picture index, and intra-prediction mode, is also supplied through the entropy coding circuitry (106) and stored in the compressed video bitstream (114).

[0027] In the encoder (100), decoder-related circuitry is also required to reconstruct pixels for the purpose of prediction. First, the prediction residual is reconstructed through inverse quantization (116) and inverse transform circuitry (118). This reconstructed prediction residual is combined with a block predictor (120) to generate unfiltered reconstructed pixels for the current video block.

[0028] Temporal prediction (also referred to as "inter prediction" or "motion-compensated prediction") uses reconstructed pixels from already-coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is typically signaled by one or more Motion Vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference. Additionally, if multiple reference pictures are supported, an additional reference picture index is sent, which is used to identify which reference picture within the reference picture store the temporal prediction signal comes from.

[0029] After spatial and / or temporal prediction is performed, the intra / inter mode determination circuit (121) in the encoder (100) selects the best prediction mode based, for example, on a rate-distortion optimization method. Then, the block predictor (120) is subtracted from the current video block, and the resulting prediction residual is de-correlated using a conversion circuit (102) and a quantization circuit (104). The resulting quantized residual coefficients are de-quantized by a de-quantization circuit (116) and de-converted by a de-conversion circuit (118) to form a reconstructed residual, and then this reconstructed residual is added back to the prediction block to form the reconstructed signal of the CU. Additional in-loop filtering (115), such as a deblocking filter, a Sample Adaptive Offset (SAO), and / or an Adaptive In-Loop Filter (ALF), may be applied to the reconstructed CU before the reconstructed CU is placed in the reference picture storage of the picture buffer (117) and used to code future video blocks. To form the output video bitstream (114), the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit (106) to be further compressed and packed to form the bitstream.

[0030] For example, deblocking filters are available in AVC and HEVC, as well as in the current version of VVC. In HEVC, an additional in-loop filter, referred to as SAO (Sample Adaptive Offset), is defined to further improve coding efficiency. In the current version of the VVC standard, another in-loop filter, referred to as ALF (Adaptive Loop Filter), is being actively researched and has a good chance of being included in the final standard.

[0031] These in-loop filter operations are optional. Performing these operations helps improve coding efficiency and visual quality. These operations can also be turned off as a decision made by the encoder (100) to reduce computational complexity.

[0032] It should be noted that while intra prediction is generally based on unfiltered reconstructed pixels, inter prediction is based on filtered reconstructed pixels when these filter options are turned on by the encoder (100).

[0033] FIG. 2 is a block diagram presenting an exemplary video decoder (200) that can be used in conjunction with many video coding standards. This decoder (200) is similar to the reconstruction-related section residing in the encoder (100) of FIG. 1. In the decoder (200) ( FIG. 2), the incoming video bitstream (201) is first decoded through entropy decoding (202) to derive quantized coefficient levels and prediction-related information. Then, the quantized coefficient levels are processed through inverse quantization (204) and inverse transformation (206) to obtain a reconstructed prediction residual. A block predictor mechanism implemented in an intra / inter mode selector (212) is configured to perform intra prediction (206) or motion compensation (210) based on the decoded prediction information. By summing the reconstructed prediction residual from the inverse transformation (206) and the prediction output generated by the block predictor mechanism using a summer (214), a set of unfiltered reconstructed pixels is obtained.

[0034] The reconstructed block may also pass through an in-loop filter (209) before being stored in a picture buffer (213) which functions as a reference picture storage. Then, the reconstructed video in the picture buffer (213) may be output to drive a display device and may also be used to predict future video blocks. When the in-loop filter (209) is turned on, a filtering operation is performed on these reconstructed pixels to derive a final reconstructed video output (222).

[0035] In video coding standards such as HEVC, blocks can be partitioned based on quaternary trees. In newer video coding standards, such as the current VVC, more partitioning methods may be used, and a single Coding Tree Unit (CTU) can be divided into CUs based on quaternary, binary, or ternary trees to adapt to various local characteristics. The separation of CUs, Prediction Units (PUs), and Transform Units (TUs) does not exist in most coding modes in the current VVC, and each CU is always used as a basic unit for both prediction and transformation without additional partitions. However, in some specific coding modes, such as intra-subpartition coding mode, each CU can still contain multiple TUs. In a multi-type tree structure, a single CTU is first partitioned by a quaternary tree structure. Next, each quad-tree leaf node can be further partitioned by binary and ternary tree structures.

[0036] FIG. 3 shows five partitioning types used in the current VVC, namely quaternary partitioning (301), horizontal binary partitioning (302), vertical binary partitioning (303), horizontal ternary partitioning (304), and vertical ternary partitioning (305). In a situation where a multi-type tree structure is utilized, one CTU is first partitioned by a quaternary-tree structure. Then, each quaternary-tree leaf node can be further partitioned by binary and ternary tree structures.

[0037] Using one or more of the exemplary block partitions (301, 302, 303, 304, or 305) of FIG. 3, spatial prediction and / or temporal prediction can be performed using the configuration shown in FIG. 1. Spatial prediction (or "intra prediction") uses pixels from samples of already-coded neighboring blocks (referred to as reference samples) in the same video picture / slice to predict the current video block. Spatial prediction reduces spatial redundancy inherent in the video signal.

[0038] In newer video coding standards, such as the current VVC, a new coding tool, Luma Mapping with Chroma Scaling (LMCS), has been added. LMCS is added as a new coding tool applied before loop filters (e.g., de-blocking filter, SAO, and ALF).

[0039] Generally, LMCS has two main modules: the first is in-loop mapping of the luminance component based on adaptive piecewise linear models, and the second is luminance-dependent chroma residual scaling.

[0040] FIG. 4 illustrates a modified decoding process to which LMCS is applied. In FIG. 4, specific blocks represent decoding modules performed in a mapped domain, which include entropy decoding (401), inverse quantization (402), inverse transform (403), luminance intra prediction (404), and luminance sample reconstruction (405) (i.e., reconstructed luminance sample (Y' recon To generate ) luma prediction samples (Y' pred ) and Luminous residual samples (Y' res It includes adding ). Other specified blocks represent decoding modules performed in the original domain (i.e., unmapped domain), which include motion-compensated prediction (409), chroma intra prediction (412), and chroma sample reconstruction (413) (i.e., reconstructed chroma samples (C recon To generate chroma prediction samples (C pred ) and chroma residual samples (C resIt includes adding ), and an in-loop filter process (407) (which includes deblocking, SAO, and ALF). An additional group of blocks represents new operation modules introduced by the LMCS, which include forward mapping (410) and inverse mapping (or backward mapping) (406) of luminance samples, and chroma residual scaling (411). Additionally, as shown in FIG. 4, all reference pictures stored in the decoded picture buffer (DPB) (408) (for luminance) and the decoded picture buffer (DPB) (415) (for chroma) are in the original domain.

[0041] In-loop mapping in LMCS aims to adjust the dynamic range of the input signal to improve coding efficiency. In conventional LMCS designs, the in-loop mapping of luminance samples is built upon two mapping functions: a forward mapping function (FwdMap) and a corresponding inverse mapping function (InvMap). The forward mapping function is signaled from the encoder to the decoder using a single fragment-by-fragment linear model with 16 equivalent fragments. The inverse mapping function can be derived directly from the forward mapping function and therefore does not need to be signaled.

[0042] The parameters of the luminance mapping model are signaled at the slice level. A presence flag is signaled first to indicate whether the luminance mapping model should be signaled for the current slice. If the luminance mapping model is present in the current slice, the corresponding fragment-specific linear model parameters are further signaled. Based on the fragment-specific linear model, the dynamic range of the input signal is partitioned into 16 segments of equal size in the original domain, and each segment is mapped to a corresponding segment. For a given segment in the original domain, the corresponding segment in the mapped domain may have the same size or a different size. The size of each segment in the mapped domain is indicated by the number of codewords in that segment (i.e., the mapped sample values). For each segment in the original domain, linear mapping parameters can be derived based on the number of codewords within the corresponding segment in the mapped domain. For example, if the input is 10-bit deep, each of the 16 segments in the original domain has 64 pixel values, and if each of the segments in the mapped domain also has 64 codewords assigned to it, this represents a simple 1-to-1 mapping (i.e., a mapping where each sample value is not changed). The number of signaled codewords for each segment in the mapped domain is used to calculate the scaling factor and, accordingly, to adjust the mapping function for that segment. Additionally, at the slice level, another LMCS control flag is signaled to enable / disable the LMCS for the slice.

[0043] For each segment, the corresponding segment-specific linear model is defined as described in the box immediately below this paragraph.

[0044]

[0045] As exemplified in Fig. 4, there is a need for operations in two different domains during the LMCS process. For each CU ("Inter CU") coded via the inter-prediction mode, the motion-compensated prediction is performed in the original domain. However, because the reconstruction of the luminance component (i.e., adding the luminance prediction samples and the luminance residual samples) is performed in the mapped domain, the motion-compensated luminance prediction (Y pred ) is a value (Y' in the domain mapped from the original domain through the forward mapping function (410). pred As ) to, Y' pred Before being used for pixel reconstruction (405), it needs to be mapped (i.e., Y' pred = FwdMap(Y pred )). On the other hand, for each CU ("Intra CU") coded through intra-prediction mode, Y' pred Considering that intra prediction (404) is performed in the mapped domain (as shown in FIG. 4) before being used for pixel reconstruction (405), mapping of the prediction samples is not necessary. Finally, the reconstructed luminance samples (Y' recon After generating ), the reconstructed luminance sample (Y' recon Before proceeding to the Luma DPB (408), the value in the original domain (Y recon A reverse mapping function (406) is applied to switch back to ) (i.e., Y recon = InvMap(Y' recon Unlike the forward mapping (410) of the predicted samples, which needs to be applied only to the inter-CUs, the reverse mapping (406) of the reconstructed samples needs to be applied to both the inter-CUs and the intra-CUs.

[0046] In summary, on the decoder side, the in-loop luminance mapping of the current LMCS is the luminance prediction samples (Y predIf necessary, this is performed by first converting to the mapped domain (Y'pred = FwdMap(Ypred)). Then, the mapped predicted samples are added to the decoded luminous residuals in the mapped domain to form reconstructed luminous samples (Y'recon = Y'pred + Y'res). Finally, inverse mapping is applied to convert the reconstructed luminous samples (Y'recon) back to the original domain (Yrecon = InvMap(Y'recon)). On the encoder side, since the luminous residuals are coded in the mapped domain, they arise as the difference between the mapped original luminous samples and the mapped predicted luminous samples (Y'res = FwdMap(Yorg) - FwdMap(Ypred)).

[0047] Luma-dependent chroma residual scaling, the second stage of LMCS, is designed to compensate for the interaction of quantization precision between the lumina signal and its corresponding chroma signals when in-loop mapping is applied to the lumina signal. Whether chroma residual scaling is enabled or disabled is also signaled in the slice header. If lumina mapping is enabled, and if dual-tree partitioning of the lumina and chroma components is disabled for the current slice, an additional flag is signaled to indicate whether or not lumina-dependent chroma residual scaling is applied. When lumina mapping is not used, or when dual-tree partitioning is enabled for the current slice, lumina-dependent chroma residual scaling is always disabled. Additionally, chroma residual scaling is always disabled for CUs containing four or fewer chroma samples.

[0048] For both intra-CU and inter-CU, the scaling parameters used to scale the chroma residuals depend on the mean of the corresponding mapped luminance prediction samples. The scaling parameters are derived as described in the box immediately below this paragraph.

[0049]

[0050] Figure 4 also illustrates the computation of the mean of the luminance prediction samples for luminance-dependent chroma residual scaling. For inter-CUs, the forward-mapped luminance prediction (Y' pred These chroma residuals (C res To derive ), scaled chroma residuals (C resScale ) is supplied to the chroma residual scaling (411) together with ), and the chroma residuals (C res ) are reconstructed chroma values ​​(C recon To derive ) chroma predictions (C pred It is supplied to chroma reconstruction (413) along with ). For intra CUs, intra prediction (404) is Y' already in the mapped domain. pred Generate , and Y' pred It is supplied to the chroma residual scaling (411) in a manner similar to that for the inter CUs.

[0051] Unlike sample-based luminance mapping, C ScaleInv is fixed for the entire chroma CU. C ScaleInv Given , chroma residual scaling is applied as described in the box immediately below this paragraph.

[0052]

[0053] In newer video coding standards, such as the current VVC, new coding tools have been introduced, some examples of which are Bidirectional Optical Flow (BDOF), Decoder-side Motion Vector Refinement (DMVR), Combined Inter and Intra Prediction (CIIP), Affine Mode, and Prediction Refinement with Optical Flow (PROF).

[0054] In the current VVC, Bidirectional Optical Flow (BDOF) is applied to refine the prediction samples of bidirectionally predicted coding blocks.

[0055] FIG. 5 is an example of a BDOF process. BDOF is a sample-by-sample motion refinement performed on block-based motion-compensated predictions when bidirectional prediction is used. Motion refinement (v) of each 4×4 sub-block (501) x , v y ) is calculated by minimizing the difference between the reference picture list 0 (LO) and reference picture list 1 (L1) prediction samples (502 and 503) after BDOF is applied within one 6×6 window (Ω) around the sub-block.

[0056] Specifically, motion refinement (v x , v y The value of ) is derived as described in the box immediately below this paragraph.

[0057]

[0058] The values ​​S1, S2, S3, S5, and S6 in the box immediately above are also calculated as described in the box immediately below this paragraph.

[0059]

[0060] The values ​​I from the box directly above (k) (i,j) are sample values ​​at coordinates (i,j) of the predicted signal within list k (k = 0,1), which are generated at medium high precision (i.e., 16-bit), and the values ​​are and are the horizontal gradient and vertical gradient of the corresponding sample, obtained by directly calculating the difference between two neighboring samples. The values ​​are and is calculated as explained in the box immediately below this paragraph.

[0061]

[0062] Based on the derived motion refinement described in the previous box, the final bidirectional prediction samples of the CU are calculated by interpolating L0 / L1 prediction samples along the motion trajectory based on the optical flow model, as shown in the box immediately below this paragraph.

[0063]

[0064] Based on the bit-depth control method described above, it is guaranteed that the maximum bit-depth of the intermediate parameters of the entire BDOF process does not exceed 32 bits, and that the largest input for multiplication is within 15 bits, that is, that one 15-bit multiplier is sufficient for the BDOF implementation.

[0065] DMVR is a bilateral prediction technique used to merge blocks into two initial signaled MVs that can be further refined by using bilateral matching prediction.

[0066] Specifically, in DMVR, two-sided matching is used to derive motion information of the current CU by finding the best match between two blocks along the motion trajectory of the current CU in two different reference pictures. The cost function used in the matching process is a row-subsampled Sum of Absolute Difference (SAD). After the matching process is performed, the refined MVs are used for motion compensation in the prediction stage (predicting temporal motion vectors for subsequent pictures), and the unrefined MVs are used for motion vector prediction between the motion vector of the current CU and the motion vectors of the current CU's spatial neighbors.

[0067] Under the assumption of a continuous motion trajectory, motion vectors (MV0 and MV1) pointing to two reference blocks will be proportional to the temporal distances between the current picture and the two reference pictures, namely TD0 and TD1. As a special case, if the current picture is temporally between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, the bilateral matching becomes a mirror-based bi-directional MV.

[0068] In current VVCs, inter- and intra-prediction methods are used in hybrid video coding schemes, where each PU is allowed to select only inter-prediction or intra-prediction to utilize correlation in the temporal or spatial domain (whereas they are never allowed to select both). However, as pointed out in previous literature, residual signals generated by inter-predicted blocks and intra-predicted blocks can exhibit very different characteristics. Therefore, if the two types of prediction can be combined in an efficient manner, a single more accurate prediction can be expected to reduce the energy of the prediction residuals and thus improve coding efficiency. Additionally, in actual video content, the motion of moving objects can be complex. For example, there may exist regions containing both old content (e.g., objects included in previously coded pictures) and emerging new content (e.g., objects excluded from previously coded pictures). In such scenarios, neither inter-prediction nor intra-prediction can provide a single accurate prediction for the current block.

[0069] To further improve prediction efficiency, Combined Inter and Intra Prediction (CIIP), which combines the intra prediction and inter prediction of a single CU coded by merge mode, is adopted in the VVC standard. Specifically, for each merged CU, an additional flag is signaled to indicate whether CIIP is enabled for the current CU. If the flag is equal to 1, CIIP applies only planar mode to generate intra-predicted samples of the luminance and chroma components. Additionally, equal weights (i.e., 0.5) are applied to average the inter-predicted samples and intra-predicted samples as the final prediction samples of the CIIP CU.

[0070] VVC also supports affine modes for motion-compensated prediction. In HEVC, only translational motion models are applied for motion-compensated prediction. In the real world, there are many types of motion, such as zoom in / out, rotation, perspective motion, and other irregular motions. In VVC, affine motion-compensated prediction is applied by signaling a single flag for each intercoding block to indicate whether a translational motion or affine motion model is applied for intercoding. In the current VVC design, two affine modes, including a 4-parameter affine mode and a 6-parameter affine mode, are supported for a single affine coding block.

[0071] The 4-parameter affine model has the following parameters: two parameters for translational motion in the horizontal and vertical directions, one parameter for zoom motion in both directions, and one parameter for rotational motion. The horizontal zoom parameter is the same as the vertical zoom parameter. The horizontal rotation parameter is the same as the vertical rotation parameter. To achieve better accommodation of motion vectors and affine parameters, in VVC, these affine parameters are converted into two MVs (also referred to as Control Point Motion Vectors, CPMVs) located at the top-left corner and top-right corner of the current block. The affine motion field of the block is described by the two control point MVs (V0, V1).

[0072] Based on control point motion, the motion field (v) of a single affine-coded block x , v y) is calculated as explained in the box immediately below this paragraph.

[0073]

[0074] The 6-parameter affine mode has the following parameters: two parameters for translational motion in the horizontal and vertical directions, one parameter for zoom motion and one parameter for rotational motion in the horizontal direction, and one parameter for zoom motion and one parameter for rotational motion in the vertical direction. The 6-parameter affine motion model is coded into three MVs in three CPMVs.

[0075] Three control points of a 6-parameter affine block are located at the top-left, top-right, and bottom-left corners of the block. Motion at the top-left control point is related to translational motion, motion at the top-right control point is related to rotation and zoom motion in the horizontal direction, and motion at the bottom-left control point is related to rotation and zoom motion in the vertical direction. Compared to a 4-parameter affine motion model, the rotation and zoom motion in the horizontal direction of the 6-parameter model may not be the same as the rotation and zoom motion in the vertical direction.

[0076] Assuming (V0, V1, V2) are the MVs of the top-left, top-right, and bottom-left corners of the current block, the motion vector (v) of each sub-block x , v y ) is derived using three MVs at control points as described in the box immediately below this paragraph.

[0077]

[0078] To improve affine motion compensation precision, Prediction Refinement with Optical Flow (PROF), which refines sub-block-based affine motion compensation based on an optical flow model, is currently being studied in the current VVC. Specifically, after performing sub-block-based affine motion compensation, the luminance prediction sample of a single affine block is modified by a single sample refinement value derived based on the optical flow equation. In detail, the operations of PROF can be summarized into the following four steps.

[0079] In Step 1, sub-block-based affine motion compensation is performed to generate a sub-block prediction I(i,j) using sub-block MVs derived from the previous box for the 4-parameter affine model and the previous box for the 6-parameter affine model.

[0080] In Step 2, g are the spatial gradients of each prediction sample. x (i,j) and g y (i,j) is calculated as described in the box immediately below this paragraph.

[0081]

[0082] In Step 2, in order to calculate the gradients, one additional row / column of prediction samples needs to occur on each side of a sub-block. To reduce memory bandwidth and complexity, samples on the extended boundaries are copied from the nearest integer pixel location in the reference picture to avoid additional interpolation processes.

[0083] In Step 3, the Luma prediction refinement value is calculated as described in the box immediately below this paragraph.

[0084]

[0085] Additionally, in the current PROF design, after adding precision refinement to the original prediction sample, a clipping operation is performed as a fourth step to clip the value of the refined precision sample to within 15 bits, as described in the box immediately below this paragraph.

[0086]

[0087] Since the affine model parameters and pixel positions relative to the center of the sub-block do not change from one sub-block to another, △v(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. If △x and △y are the horizontal and vertical offsets from the sample position (i,j) to the center of the sub-block to which the sample belongs, △v(i,j) can be derived as described in the box immediately below this paragraph.

[0088]

[0089] Based on the affine sub-block MV derivation equations in the preceding boxes for the 4-parameter affine model and the 6-parameter affine model, the MV difference △v(i,j) can be derived as described in the box immediately below this paragraph.

[0090]

[0091] According to the current LMCS design, chroma residual samples are scaled based on their corresponding luminance prediction samples. When newer coding tools are enabled for inter-CU, the luminance prediction samples used to scale chroma residual samples through LMCS in these inter-CUs are obtained at the end of sequential applications of these newer coding tools.

[0092] FIG. 6 is a flowchart illustrating the workflow of chroma residual scaling in LMCS when DMVR, BDOF, and CIIP are all enabled. Outputs from the luminance L0 prediction value (601) and L1 prediction value (602) are sequentially fed to DMVR (603) and BDOF (604), and the resulting luminance inter prediction value (621) is fed to the average (606) together with the luminance intra prediction value (622) from the luminance intra prediction (605) to generate an averaged luminance prediction value (623), and the averaged luminance prediction value (623) is fed to the chroma residual scaling (607) together with the chroma residuals (608), so that the chroma residual scaling (607), chroma prediction (610), and chroma reconstruction (609) can work together to produce a final result.

[0093] Current LMCS designs present three challenges to the video decoding process. First, mappings between different domains require additional computational complexity and on-chip memory. Second, the fact that the derivation of luminance and chroma scaling factors uses different luminance prediction values ​​introduces additional complexity. Third, the interaction between LMCS and newer coding tools introduces latency into the decoding process.

[0094] First, in current LMCS designs, both the original domain and the reconstructed samples in the mapped domain are used in various decoding modules. As a result, these samples often need to be converted from one domain to another between different decoding modules, which can lead to both higher computational complexity and more on-chip memory.

[0095] Specifically, for Intra mode, CIIP mode, and IBC mode, mapped domain reference samples from neighboring reconstructed regions of a current CU are used to generate prediction samples. However, for Inter mode, motion-compensated prediction is performed using original domain reconstructed samples of temporal reference pictures as references. The reconstructed samples stored in the DPB are also in the original domain.

[0096] For example, for inter-CUs, since the luminance reconstruction operation (i.e., adding the predicted samples and residual samples together) is performed in the mapped domain, inter-predicted luminance samples generated in the native domain need to be converted to the mapped domain before they are used for luminance sample reconstruction. In another example, for both intra-CUs and inter-CUs, inverse mapping (or reverse mapping) is always applied to convert the reconstructed luminance samples from the mapped domain to the native domain before storing them in the DPB. This design not only increases computational complexity due to additional forward / inverse mapping operations but also requires more on-chip memory to maintain multiple versions of the reconstructed samples.

[0097] In actual hardware implementations, the forward mapping function (FwdMap) and the inverse (or reverse) mapping function (InvMap) can be implemented using a Look-Up Table (LUT) or computed on-the-fly. If a LUT-based solution is used, the possible output elements from the functions (FwdMap, InvMap, and cScaleInv) can be pre-computed and pre-stored as LUTs, and then used for the luminance mapping and chroma residual scaling operations of all CUs within the current slice. Assuming the input video is 10-bit, within each of the LUTs for FwdMap and InvMap, 2 10 = There are 1024 elements, and each element within the LUTs has 10-bits. Therefore, the total storage for the forward and inverse luma mapping LUTs is equivalent to 2 * 1024 * 10 = 20480 bits = 2560 bytes. On the other hand, chroma scaling parameters (C ScaleInv To derive ), a single 16-entry LUT table (cScaleInv) needs to be maintained in the encoder and decoder, and each chroma scaling parameter is stored in 32 bits. Accordingly, the memory size used to store the LUT (cScaleInv) is equivalent to 16 * 32 = 512 bits = 64 bytes. The difference between 2560 and 64 shows the scale of additional on-chip memory required by forward and inverse (reverse) mapping operations.

[0098] Furthermore, in newer video coding standards such as the current VVC, both intra-prediction and deblocking filters use reconstructed samples from preceding neighboring blocks. Therefore, one additional row of reconstructed samples within the width of the current picture / slice needs to be maintained in a buffer (which is also known as a "line-buffer" in video coding). The reconstructed samples in the line-buffer are used as references for the intra-prediction and deblocking operations of CUs located in the first row within a single CTU. According to conventional LMCS designs, the intra-prediction and deblocking filters use reconstructed samples in different domains. Consequently, additional on-chip memory is required to store both the original domain and the mapped domain reconstructed samples, which can be approximately double the size of the line-buffer.

[0099] One implementation choice to avoid doubling the line-buffer size is to perform domain mapping operations on-the-fly. However, this comes at the expense of a significant increase in computational complexity.

[0100] Therefore, current designs of LMCS will require additional computational complexity and on-chip memory due to the required mappings between different domains.

[0101] Secondly, although both methods for deriving luminance and chroma scaling factors in the current design of LMCS use luminance prediction sample values ​​to derive corresponding scaling factors, there are differences between their corresponding operations.

[0102] For luminal residual scaling, scaling factors are derived on a sample-by-sample basis, allowing each luminal residual sample to have its own scaling factor. However, for chroma residual scaling, the scaling factor is fixed for the entire CU, meaning that all chroma residual samples within the CU share the same scaling factor calculated based on the mean of the mapped luminal prediction samples.

[0103] Additionally, two different LUTs are used to calculate the scaling factors of the luminance and chroma residuals. Specifically, the input to the luminance LUT is the mapping model segment index of the original luminance predicted sample values, whereas the input to the chroma LUT is the mapping model segment index of the mean value of the mapped luminance predicted samples.

[0104] These differences introduce additional complexity into the coding process, and a harmonized approach to deriving luminance and chroma scaling factors is desirable.

[0105] Thirdly, for chroma residual scaling in the current design of LMCS, newer coding tools such as all three modules of DMVR, BDOF, and CIIP can be called sequentially to generate luminance prediction samples, which are then used to determine the scaling factor for chroma residuals. Given the high computational complexity of the three modules, waiting for their successful completion before performing chroma residual scaling in LMCS can result in significant latency for decoding chroma samples. For affine CUs, the PROF process can also have latency issues because each affine CU can perform the PROF process and subsequently perform LMCS, which can also lead to latency issues for decoding chroma samples.

[0106] Furthermore, in the current design of LMCS, unnecessary clipping is performed during the chroma residual scaling factor derivation process, which further increases computational complexity and the additional requirements for on-chip memory.

[0107] The present disclosure aims to address or mitigate these challenges presented by the current design of LMCS, and more specifically, the present disclosure discusses ways to reduce the complexity of LMCS for hardware codec implementation while maintaining coding gains.

[0108] Instead of using the existing LMCS framework, which transforms prediction / reconstruction samples through mapping operations, a new method referred to as Prediction-Dependent Residual Scaling (PDRS) is proposed to directly scale prediction residuals without sample mapping. The proposed method can achieve similar effectiveness and coding efficiency to LMCS but has much lower implementation complexity.

[0109] In the PDRS procedure, as illustrated in FIG. 7, a luminance prediction sample is obtained to decode a luminance residual sample (701), a scaling factor is derived using the luminance prediction sample (702), the scaling factor is used to scale the luminance residual sample (703), and the reconstructed luminance sample is calculated by adding the luminance prediction sample and the scaled luminance residual sample (704).

[0110] Unlike conventional LMCS methods, which directly convert predicted / reconstructed luminance samples into a mapped domain before calculating the luminance prediction residuals, the proposed method of the PDRS procedure derives luminance prediction residual samples from the original domain in the same manner as in the normal prediction process without any mapping operations, followed by a scaling operation for the luminance prediction residuals. The scaling of the luminance prediction residuals depends on the corresponding luminance prediction sample values ​​and the fragment-by-fragment linear model. As a result, the forward and inverse luminance mapping operations in current LMCS designs can be completely discarded, and all relevant prediction and reconstruction samples are retained in the original sample domain during the decoding process. Based on the aforementioned features, the proposed method is named prediction-dependent residual scaling.

[0111] FIG. 8 is a flowchart illustrating the workflow of the decoding process when the PDRS procedure is applied in the LMCS process. FIG. 8 illustrates the elimination of the need for mapping between different domains. Now, with the exception of residual decoding modules (e.g., entropy decoding (801), inverse quantization (802), and inverse transform (803)), all other decoding modules (including intra- and inter-prediction (804, 809, 812, and 816), reconstruction (806 and 813), and all in-loop filters (807 and 814)) are operating in the original domain. Specifically, to reconstruct the luminance samples, the proposed method in the PDRS procedure only applies to the luminance-predicted residual samples (Y res ) are de-scaled back to their original amplitude levels, and then, these are the luminance prediction samples (Y pred There is only a need to add to ).

[0112] By utilizing the PDRS procedure, forward and inverse luminance sample mapping operations in conventional LMCS designs are completely eliminated. This not only reduces computational complexity but also decreases the size of potential storage for storing LMCS parameters. For example, when a LUT-based solution is used to implement luminance mapping, the storage previously used to store two mapping LUTs (FwdMap[] and InvMap[]) (approximately 2560 bytes) is no longer required in the proposed method. Furthermore, unlike conventional luminance mapping methods that require storing reconstructed luminance samples in both the original domain and the mapped domain, the proposed method in the PDRS procedure generates and maintains all prediction and reconstructed samples only in the original domain. Accordingly, compared to conventional luminance mapping, the proposed method in the PDRS procedure can efficiently reduce the line-buffer size used to store reconstructed samples for intra-prediction and deblocking by half.

[0113] According to one or more embodiments of the PDRS procedure, the luminance prediction sample and the luminance residual sample originate from a single collocated position in the luminance prediction block and its associated residual block.

[0114] According to one or more embodiments of the PDRS procedure, deriving a scaling factor using a luminance prediction sample comprises dividing the entire range of possible luminance prediction sample values ​​into a plurality of luminance prediction sample segments, calculating a scaling factor for each of the plurality of luminance prediction sample segments based on a pre-defined fragment linear model, and determining a scaling factor of the luminance prediction sample based on the scaling factors of the plurality of luminance prediction sample segments.

[0115] In one example, determining the scaling factor of a luminance prediction sample based on the scaling factors of a plurality of luminance prediction sample segments includes assigning the luminance prediction sample to one of the plurality of luminance prediction sample segments, and calculating the scaling factor of the luminance prediction sample as the scaling factor of the assigned luminance prediction sample segment.

[0116] In this example, multiple luminance prediction sample segments include 16 segments in a pre-defined 16-fragment LUT table (scaleForward), and a pre-defined fragment linear model for calculating one scaling factor for each of the multiple luminance prediction sample segments includes 16 values ​​corresponding to the 16 segments in the pre-defined LUT table (scaleForward).

[0117] In the same example, the scaling factor is calculated based on the assigned luma prediction sample segments, as described in the box immediately below this paragraph.

[0118]

[0119] In the same example, the luminous scaling factor (Scale Y Given ), the Luma residual sample scaling method can be applied as described in the box immediately below this paragraph.

[0120]

[0121] The motivation behind this example is that forward mapping in the current LMCS is based on a single fragment-by-fragment linear model. If both the original luminance sample and the predicted luminance sample are located on the same fragment (i.e., the same segment defined by two pivot points (InputPivot[i] and InputPivot[i+1])), then the two forward mapping functions of the original luminance sample and the predicted luminance sample become exactly the same. Accordingly, this leads to Y'res = FwdMap(Yorg) - FwdMap(Ypred) = FwdMap(Yorg - Ypred) == FwdMap(Yres). By applying inverse mapping to both sides of this equation, the corresponding decoder-side reconstruction operation can be expressed as Yrecon = Ypred + InvMap(Y'res).

[0122] In other words, in a situation where both the original luminance sample and the luminance prediction sample are located in the same fragment, the luminance mapping method in LMCS can be achieved through a single residual scaling operation in the decoding process, as implemented in this possible embodiment.

[0123] Although this conclusion is derived based on the assumption that both the original luminance sample and the predicted luminance sample are located in the same segment defined by two pivot points (InputPivot[i] and InputPivot[i+1]), such a possible implementation of this example can still be used as a simplification and / or approximation of the existing luminance mapping behavior in VVC in any case, even when the original luminance sample and the predicted luminance sample are located in different segments of the fragment-by-fragment linear model. Experimental results show that such simplification and / or approximation has little effect on coding performance.

[0124] To reiterate, this example is based on the assumption that both the original and predicted luminance sample values ​​are located in the same segment of the fragment-by-fragment linear mode. In this case, the forward / inverse mapping functions applied to the original and predicted luminance samples are identical, and therefore it is safe to calculate the corresponding residual scaling factor based only on the predicted luminance sample.

[0125] However, when the predicted samples of a CU are not sufficiently accurate (e.g., for intra-predicted CUs where samples far from the reference samples are generally predicted less accurately), the predicted samples and the original samples are often located in different segments of the fragment-by-fragment linear model. In such cases, the scaling factor derived based on the predicted sample values ​​may not be reliable in reflecting the original mapping relationship between the residual samples in the original domain (i.e., the unmapped domain) and the residual samples in the mapped domain.

[0126] Figure 9 is an example of residual mapping error resulting from using only predicted samples to derive the scaling factor. In Figure 9, the triangle-shaped filled points represent the pivot control points of different segments in the segment-by-segment linear function, the circular-shaped filled points represent the original sample values ​​and predicted sample values, and Y org and Y pred are the original samples and predicted samples in the original domain (i.e., the unmapped domain), and Y' org and Y' pred are respectively Y org and Y pred These are the mapped samples of. Y res and Y' res are the corresponding residuals in the original domain and the mapped domain when the existing sample-based luminance mapping method in VVC is applied, and Y' resScaleis a mapped residual sample derived based on the proposed luminous residual scaling method. As shown in Fig. 9, since the original sample and the predicted sample are not within the same segment of the fragment-by-fragment linear model, the scaling factor derived based on the predicted sample is the original residual in the mapped domain (i.e., Y' res The scaled residual (i.e., Y') that approximates ) resScale It may not be accurate enough to generate ).

[0127] In the second example, the assumption that both the original luminance sample value and the predicted luminance sample value are located in the same segment of the fragment-by-fragment linear mode is not required.

[0128] In this second example, instead of directly deriving the scaling factor from the segment of the segment-by-segment linear model where the luma prediction sample is located, the scaling factor is calculated as the average of the scaling factors of N neighboring segments (where N is a positive integer).

[0129] In this second example, determining the scaling factor of a luminary prediction sample based on the scaling factors of a plurality of luminary prediction sample segments includes assigning the luminary prediction sample to one of the plurality of luminary prediction sample segments, and calculating the scaling factor of the luminary prediction sample as the average of the scaling factors of a plurality of luminary prediction sample segments adjacent to the assigned luminary prediction sample segment.

[0130] More specifically, in one possible implementation of this second example, the scaling factor can be calculated based on the assigned luma prediction sample segments as described in the following steps.

[0131] 1) Pred in the original domain Y The corresponding segment index (Idx) of the fragment-by-fragment linear model to which belongs Y Find or acquire )

[0132] 2) If Y' res If ≥ 0, the luminance residual scaling factor is calculated as follows:

[0133]

[0134] 3) Otherwise (i.e., Y' res (< 0), the Luma residual scaling factor is calculated as follows

[0135]

[0136] Here, scaleForward[i](i = 0 ... 15) is a pre-defined 16-fragment LUT, which is calculated as follows

[0137] scaleForward[i] = (OrgCW << SCALE_FP_PREC) / SignaledCW[i]

[0138] Here, OrgCW and SignaledCW[i] are the number of codewords in the i-th segment within the original domain and the mapped domain, respectively, and SCALE_FP_PREC is the precision of the scaling factor.

[0140] Alternatively, in a second possible embodiment of this second example, identical to the embodiment described above, the scaling factor can be calculated based on the assigned luma prediction sample segment as described in the box immediately below this paragraph.

[0141]

[0142] The two preceding possible implementations of this second example differ only in the selection of N luma prediction sample domain value segments based on the assigned segment.

[0143] In a chroma sample reconstruction procedure, as exemplified in FIG. 10, a lumina prediction sample value for decoding both a lumina residual sample and a chroma residual sample at an input location is obtained (1001), then a lumina prediction sample associated with the lumina residual sample is obtained (1002), then a chroma prediction sample associated with the chroma residual sample is obtained (1003), the lumina prediction sample is used to derive a first scaling factor for the lumina residual sample and a second scaling factor for the chroma residual sample (1004), the first scaling factor is used to scale the lumina residual sample (1005), the second scaling factor is used to scale the chroma residual sample (1006), a reconstructed lumina sample is calculated by adding the lumina prediction sample and the scaled lumina residual sample (1007), and the reconstructed chroma sample is obtained by adding the chroma prediction sample and the scaled chroma residual sample. It is calculated (1008).

[0144] The chroma sample reconstruction procedure aims to harmonize the scaling methods of luminance and chroma residuals to achieve a more unified design.

[0145] According to one or more embodiments of the chroma sample reconstruction procedure, the luminance predicted sample value is the average of all luminance predicted samples in a coding unit (CU) containing an input location. In these embodiments, a chroma scaling derivation method is used to calculate a scaling factor for luminance residuals; more specifically, instead of deriving a single scaling factor for each luminance residual sample separately, a single shared scaling factor calculated based on the average of the luminance predicted samples is used to scale the luminance residual samples of the entire CU.

[0146] According to another embodiment of the chroma sample reconstruction procedure, the luminance prediction sample value is the average of all luminance prediction samples within a pre-defined sub-block subdivided from a coding unit (CU) containing an input location. In this embodiment, a CU is first equally partitioned into a plurality of M×N sub-blocks, and then, for each sub-block, all or partial luminance prediction samples are used to derive a corresponding scaling factor used to scale both the luminance residual and the chroma residual of the sub-block. Compared to the first method, the second method can improve the spatial precision of the estimated scaling factor because less correlated luminance prediction samples outside the sub-block are excluded from calculating the scaling factor of the sub-block. Meanwhile, the second method can also reduce the latency of reconstructing luminal and chroma residuals by considering that the scaling of luminal and chroma residuals within a single subblock can begin immediately after the luminal prediction of the subblock is completed, that is, without waiting for the entire occurrence of luminal prediction samples of the entire CU.

[0147] According to a third embodiment of the chroma sample reconstruction procedure, the luminance prediction sample domain value includes juxtaposed luminance prediction samples. In this embodiment, the luminance residual scaling method is extended to scale the chroma residuals, and different scaling factors for each chroma residual sample are derived based on the juxtaposed luminance prediction sample values.

[0148] In the preceding embodiments of the chroma sample reconstruction procedure, it is suggested to use the same LUT used to calculate the luminance scaling factor to perform scaling of the chroma residuals. In one example, to derive the CU-level scaling factor (ScaleC) for the chroma residuals, the following may follow.

[0149] 1) The mean (avg) of the luminance prediction samples (represented in the original domain) within the CU Y Calculate (indicated as)

[0150] 2) avg Y The corresponding segment index (Idx) of the fragment-by-fragment linear model to which belongs Y Find or acquire )

[0151] 3) Scale C The value of is calculated as follows

[0152] Scale C = scaleForward[Idx Y ]

[0153] Here, scaleForward[i](i = 0 ... 15) is a pre-defined 16-fragment LUT, which is calculated as follows

[0154] scaleForward[i] = (OrgCW << SCALE_FP_PREC) / SignaledCW[i]

[0155] Here, OrgCW and SignaledCW[i] are the number of codewords in the i-th segment within the original domain and the mapped domain, respectively, and SCALE_FP_PREC is the precision of the scaling factor.

[0156] The preceding example can be easily extended to a case where a scaling factor for the chroma residual is derived for each subblock of the current CU. In such a case, in the preceding Step 1, avgY is calculated as the average of the luminance prediction samples in the original domain of the subblock, while Steps 2 and 3 remain the same.

[0157] In the second chroma sample reconstruction procedure, as illustrated in FIG. 11, a plurality of luminance prediction samples are obtained by skipping a plurality of pre-defined intermediate luminance prediction stages during the luminance prediction process for a coding unit (CU) (1101), the obtained plurality of luminance prediction samples are used to derive scaling factors for chroma residual samples in the CU (1102), scaling factors are used to scale the chroma residual samples in the CU (1103), and a reconstructed chroma sample is calculated by adding the chroma prediction samples and the scaled chroma residual samples in the CU (1104).

[0158] According to one or more embodiments of the second chroma sample reconstruction procedure, pre-defined intermediate lumina prediction stages include one or more bidirectional prediction modules of decoder-side motion vector derivation (DMVR), bidirectional optical flow (BDOF), and combined inter- and intra prediction (CIIP). In these embodiments, the inter prediction samples derived prior to the DMVR, BDOF / PROF, and CIIP intra / inter combining processes are used to derive a scaling factor for the chroma residuals.

[0159] FIG. 12 is a flowchart illustrating the workflow of the LMCS decoding process in one example of this embodiment of the second chroma sample reconstruction procedure, wherein DMVR, BDOF, and CIIP are not applied to generate luminance prediction samples for chroma scaling. Here, instead of waiting for the luminance intra prediction part (1205) of DMVR (1203), BDOF (1204), and / or CIIP to be fully completed, the chroma residual scaling process (1208) may be started as soon as prediction samples (1221 and 1222) based on the initial L0 and L1 luminance predictions (1201 and 1202) become available.

[0160] In FIG. 12, in addition to the original averaging operation (1206), one additional averaging operation (1211) is required to combine the initial L0 and L1 prediction samples (1221 and 1222) prior to DMVR (1203), BDOF (1204), and / or CIIP (1205).

[0161] To reduce complexity, in the second example of this embodiment of the second chroma sample reconstruction procedure, initial L0 predicted samples can be used to derive a scaling factor for the chroma residuals.

[0162] FIG. 13 is a flowchart illustrating the workflow of the LMCS decoding process in the second example of this embodiment of the second chroma sample reconstruction procedure, wherein an initial unidirectional prediction signal is applied to generate luminance prediction samples for chroma scaling. No additional averaging operation is required in addition to the original averaging operation (1306). The initial L0 prediction samples (1321) are used to derive a scaling factor for chroma residuals before DMVR (1303), BDOF (1304), and / or CIIP (1305).

[0163] In the third example of this embodiment of the second chroma sample reconstruction procedure, one initial prediction signal (L0 or L1) is adaptively selected as a luminance prediction sample used to derive a chroma residual scaling factor. In one possible embodiment of this example, between the initial prediction signals (L0 or L1), the one in which the corresponding reference picture has a smaller Picture Order Count (POC) distance from the current picture is selected to derive the chroma residual scaling factor.

[0164] In another embodiment of the second chroma sample reconstruction procedure, it is proposed to disable only DMVR and BDOF / PROF while activating CIIP to generate inter-predicted samples used to determine the chroma residual scaling factor. Specifically, in this method, the inter-predicted samples derived before DMVR and BDOF / PROF are first averaged, and then combined with intra-predicted samples for CIIP, and finally, the combined predicted samples are used as predicted samples to determine the chroma residual scaling factor.

[0165] In another embodiment of the second chroma sample reconstruction procedure, it is proposed to disable only BDOF / PROF while maintaining DMVR and CIIP to generate prediction samples used to determine the chroma residual scaling factor.

[0166] In another embodiment of the second chroma sample reconstruction procedure, it is proposed to retain only BDOF / PROF and CIIP while disabling DMVR when deriving luminance prediction samples used to determine the chroma residual scaling factor.

[0167] Furthermore, although the methods in the preceding examples of the second chroma sample reconstruction procedure are exemplified as being designed to reduce the latency of chroma prediction residual scaling, it is worth noting that these methods can also be used to reduce the latency of luminance prediction residual scaling. For example, all these methods can also be applied to the PDRS method described in the section "Luma Mapping Based on Prediction-Dependent Residual Scaling".

[0168] According to the existing DMVR design, to reduce computational complexity, the prediction samples used for DMVR motion refinement are generated using 2-tap bilinear filters instead of default 8-tap interpolation. After the refined motion is determined, default 8-tap filters will be applied to generate the final prediction samples of the current CU. Therefore, to reduce the chroma residual decoding latency caused by DMVR, it is proposed to use the luminance prediction samples generated by the bilinear filters (the average of L0 and L1 prediction samples if the current CU is bidirectionally predicted) to determine the scaling factor of the chroma residuals.

[0169] According to a chroma residual sample reconstruction procedure, as illustrated in FIG. 14, one or more luminance prediction sample values ​​are selected from the output of a bilinear filter of decoder-side motion vector derivation (DMVR) (1401), one or more selected luminance prediction sample values ​​are adjusted to another or more luminance prediction sample values ​​having a bit depth equal to the original coding bit depth of the input video (1402), luminance prediction sample values ​​having a bit depth equal to the original coding bit depth of the input video are used to derive a scaling factor for decoding one or more chroma residual samples (1403), a scaling factor is used to scale one or more chroma residual samples (1404), and one or more chroma residual samples are reconstructed by adding one or more scaled chroma residual samples and their corresponding chroma prediction samples (1405).

[0170] In one or more embodiments of the chroma residual sample reconstruction procedure, selecting one or more luminance prediction sample values ​​from the output of the DMVR's bilinear filter includes selecting L0 and L1 luminance prediction samples from the output of the DMVR's bilinear filter.

[0171] FIG. 15 is a flowchart illustrating the workflow of the LMCS decoding process in one such embodiment of the chroma residual sample reconstruction procedure. L0 and L1 prediction samples (1521 and 1522) from the output of the bilinear filter (1512) component of the DMVR (1503) are fed to an average (1511) to derive a chroma residual scaling input (1523) to be used in chroma residual scaling (1507) to decode one or more chroma residual samples.

[0172] In these embodiments, there is a problem with bit depth. To reduce the internal storage size used by the DMVR, the intermediate L0 and L1 prediction samples generated by the DMVR's bilinear filters are at 10-bit precision. This differs from the representation bit-depth of the intermediate prediction samples of a normal bilinear prediction, which is 14-bit. Therefore, the intermediate prediction samples output from the bilinear filters cannot be directly applied to determine the chroma residual scaling factor due to the different precision.

[0173] To address this problem, it is first proposed to align the DMVR intermediate bit depth with the intermediate bit depth used for normal motion-compensated interpolation, that is, to increase the bit depth from 10 bits to 14 bits. Subsequently, the existing averaging process applied to generate a normal bidirectional prediction signal can be reused to generate corresponding prediction samples for determining the chroma residual scaling factor.

[0174] In one example of these embodiments, adjusting one or more selected luminance prediction sample values ​​to another or more luminance prediction sample values ​​having the same bit depth as the original coding bit depth of the input video comprises increasing the internal bit depth of L0 and L1 luminance prediction samples from the output of the DMVR's bilinear filter to 14-bit through left shifting, obtaining a 14-bit average luminance prediction sample value by averaging the 14-bit shifted L0 and L1 luminance prediction sample values, and converting the 14-bit average luminance prediction sample values ​​by changing the internal bit depth of the 14-bit average luminance prediction sample values ​​to the original coding bit depth of the input video through right shifting.

[0175] More specifically, in this example, the chroma scaling factor is determined by the steps described in the box immediately below this paragraph.

[0176]

[0177] In other embodiments of the chroma residual sample reconstruction procedure, selecting one or more luminance prediction sample values ​​from the output of the DMVR's bilinear filter, and adjusting one or more selected luminance prediction sample values ​​to another or more luminance prediction sample values ​​having a bit depth equal to the original coding bit depth of the input video, comprises selecting one luminance prediction sample among L0 and L1 luminance prediction samples from the output of the DMVR's bilinear filter, adjusting one selected luminance prediction sample by changing the internal bit depth of one selected luminance prediction value to the original coding bit depth of the input video through a shift, and using the adjusted luminance prediction sample as a luminance prediction sample having a bit depth equal to the original coding bit depth of the input video.

[0178] FIG. 16 is a flowchart illustrating the workflow of the LMCS decoding process in one of these other embodiments of the chroma residual sample reconstruction procedure. L0 predicted samples (1621) from the output of the bilinear filter (1612) component of the DMVR (1603) are used in chroma residual scaling (1607) to decode one or more chroma residual samples.

[0179] In one example of this other embodiment of the chroma residual sample reconstruction procedure, the chroma scaling factor is determined by shifting the luminance samples output from the bilinear filters to the original coding bit depth of the input video, as described in the box immediately below this paragraph.

[0180]

[0181] According to the second chroma residual sample reconstruction procedure, as illustrated in FIG. 17, one or more luminance reference sample values ​​are selected from reference pictures (1701), one or more selected luminance reference sample values ​​are converted into luminance sample values ​​(1702), the converted luminance sample values ​​are used to derive a scaling factor (1703), the scaling factor is used to scale one or more chroma residual samples (1704), and one or more chroma residual samples are reconstructed by adding one or more scaled chroma residual samples and their corresponding chroma prediction samples (1705).

[0182] In one or more embodiments of the second chroma residual sample reconstruction procedure, selecting one or more luminance reference sample values ​​from reference pictures and converting one or more selected luminance reference sample values ​​into luminance sample values ​​includes obtaining both L0 and L1 luminance reference sample values ​​from L0 and L1 reference pictures, and averaging the L0 and L1 luminance reference sample values ​​as converted luminance sample values.

[0183] In other embodiments of the second chroma residual sample reconstruction procedure, selecting one or more luminance reference samples from reference pictures and converting one or more selected luminance reference samples into luminance sample values ​​includes selecting one luminance reference sample value among L0 and L1 luminance reference sample values ​​from L0 and L1 reference pictures, and using one selected luminance reference sample value as a converted luminance sample value.

[0184] According to the existing LMCS design, reconstructed luminance samples neighboring the 64x64 region where the current CU is located are used to compute the chroma residual scaling factor for the CUs within the region. Additionally, a clipping operation, namely Clip1(), is applied to clip the reconstructed luminance neighbor samples to a dynamic range of internal bit depth (i.e., range [0, (1<<bit depth) - 1]) before the average is calculated.

[0185] Specifically, this method first fetches the 64 left-neighboring luminal samples and 64 top-neighboring luminal samples of the corresponding 64 x 64 region to which the current CU belongs, then calculates the average of the left and top neighboring samples, i.e., avgY, finds the segment index (Y1dx) of avgY in the LMCS fragment-by-fragment linear model, and finally the chroma residuals (C ScaleInv = cScaleInv[Y1dx]) derives.

[0186] Specifically, in the current VVC draft, the method for deriving the corresponding average luma is described as follows, where the application of the Clip1() operation is shown with prominent font type and size.

[0187] The following steps are applied to derive the variable (varScale).

[0188] The variable (invAvgLuma) is derived as follows.

[0189] - An array (recLuma[i]) and a variable (cnt) with i=0..(2 * sizeY - 1) are derived as follows.

[0190] - The variable (cnt) is set to be equal to 0.

[0191] - When availL is equivalent to true, the array (recLuma[i]) with i = 0..sizeY - 1 is set to be equal to currPic[xCuCb - 1][Min(yCuCb + i, pic_height_in_luma_samples - 1)] with i = 0..sizeY - 1, and cnt is set to be equal to sizeY.

[0192] - If availT is equivalent to TRUE, the array (recLuma[cnt + i]) with i = 0..sizeY - 1 is set to be equal to currPic[Min(xCuCb + i, pic_width_in_luma_samples - 1)][yCuCb - 1] with i = 0..sizeY - 1, and cnt is set to be equal to (cnt + sizeY).

[0193] - The variable (invAvgLuma) is derived as follows.

[0194] - If cnt is greater than 0, the following applies.

[0195]

[0196] - Otherwise (if cnt is equal to 0), the following applies.

[0197] invAvgLuma = 1 << (bit depth - 1)

[0198] However, during the reconstruction process, after adding the predicted samples to the residual samples of a single CU, the resulting sample values ​​are already clipped to the dynamic range of the internal bit depth. This means that all neighboring reconstructed luminance samples around the current 64x64 region are guaranteed to be within the range of the internal bit depth. Therefore, their average, i.e., avgY, cannot exceed this range. Consequently, the existing clipping (i.e., Clip1()) is unnecessary for calculating the corresponding chroma residual scaling factor. To further reduce the complexity and memory requirements of the LMCS design, it is proposed to remove the clipping operation when calculating the average of neighboring reconstructed luminance samples to derive the chroma residual scaling factor.

[0199] According to a first embodiment of the present application, as illustrated in FIG. 18, during the decoding of the CU, a plurality of reconstructed luminance samples are selected from a first predetermined region (the first predetermined region is adjacent to a second predetermined region where the CU is located) (1801), the average of the plurality of reconstructed luminance samples is calculated (1802), and when deriving a chroma residual scaling factor for decoding the CU, the average of the plurality of reconstructed luminance samples is used directly without any clipping (1803).

[0200] In one or more embodiments of the first embodiment of the present application, the average of a plurality of reconstructed luma samples is the arithmetic average of a plurality of reconstructed luma samples.

[0201] In one or more embodiments of the first embodiment of the present application, when deriving a chroma residual scaling factor for decoding CU, directly using the mean of a plurality of reconstructed luminance samples without any clipping, identifies a segment index for the mean in a pre-defined segment-by-segment linear model, and It includes deriving a chrome residual scaling factor for decoding CU based on the slope of the linear model of the segment.

[0202] In one or more embodiments of the first embodiment of the present application, a plurality of reconstructed luminance samples within a first predetermined region are generated by performing the following: generating luminance prediction samples and luminance residual samples within the first predetermined region, adding the luminance residual samples to the luminance prediction samples, and clipping the added luminance samples to a dynamic range of coding bit depth.

[0203] In one or more embodiments of the first embodiment of the present application, a plurality of reconstructed luma samples are a plurality of forward-mapped inter-luma reconstructed samples.

[0204] In one or more embodiments of the first embodiment of the present application, the second predetermined region is a 64x64 region where the CU is located.

[0205] In one example, as illustrated in FIG. 19, the first predetermined region may include upper neighbor samples within a 1x64 region (1902) immediately above the second predetermined region (1904). Alternatively or additionally, the first predetermined region may include left neighbor samples within a 64x1 region (1903) immediately to the left of the second predetermined region (1904). Alternatively or additionally, the first predetermined region may include a 1x64 region immediately above the second predetermined region, and a 64x1 region immediately to the left of the second predetermined region.

[0206] In one or more embodiments, the functions described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored on a computer-readable medium as one or more instructions or code, or transmitted through a computer-readable medium, and may be executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to media of the same type as data storage media, or may include communication media including any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol, for example. In this way, computer-readable media may generally correspond to (1) non-transient types of computer-readable storage media or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for the implementation of the embodiments described in this application. Computer program products may include computer-readable media.

[0207] Furthermore, the methods described above may be implemented using a device comprising one or more circuits, such circuits including Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components. The device may use these circuits in combination with other hardware or software components to perform the methods described above. Each module, sub-module, unit, or sub-unit disclosed above may be implemented at least partially using one or more circuits.

[0208] The present disclosure may include dedicated hardware embodiments, such as application-specific integrated circuits, programmable logic arrays, and other hardware devices. Hardware embodiments may be configured to implement one or more of the methods described herein. Examples that may include devices and systems of various embodiments may broadly include various electronic systems and computing systems. One or more examples described herein may implement functions using two or more specific interconnected hardware modules or devices having related control and data signals that can be transmitted between and through the modules, or as parts of an application-specific integrated circuit. Accordingly, the devices or systems disclosed herein may encompass software, firmware, and hardware embodiments. The terms “module,” “sub-module,” “circuit,” “sub-circuit,” “circuitry,” “sub-circuitry,” “unit,” or “sub-unit” may include (shared, private, or group) memory that stores code or instructions that can be executed by one or more processors. A module referred to herein may include one or more circuits that may or may not have stored code or instructions. A module or circuit may include one or more connected components.

[0209] Other embodiments of the present invention will be apparent to those skilled in the art from consideration of the specifications and practices of the present invention disclosed herein. This application is intended to encompass any variations, uses, or modifications of the present invention in accordance with the general principles of the present invention, including such deviations from the disclosure as being within the scope of practices known or common to the art. The specifications and examples are intended to be considered merely as illustrative, and the true scope and spirit of the present invention are indicated by the following claims.

[0210] It will be understood that the present invention is not limited to the exact examples described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope of the invention. The scope of the invention is intended to be limited only by the appended claims.

Claims

Claim 1 As a method for video decoding, the above method is, Receiving a bitstream containing a Coding Unit (CU); Obtaining the average of a plurality of reconstructed luma samples adjacent to a predetermined region where the above-mentioned coding unit (CU) is located; and A method for video decoding characterized by including deriving a chroma residual scaling factor for decoding the CU by directly using the average of the plurality of reconstructed luminance samples without any clipping. Claim 2 A method for video decoding according to claim 1, characterized in that the average of the plurality of reconstructed luma samples is the arithmetic average of the plurality of reconstructed luma samples. Claim 3 In claim 1, deriving the chroma residual scaling factor for decoding the CU by directly using the average of the plurality of reconstructed luminance samples without any clipping is, Identifying the segment index for the mean in a pre-defined piecewise linear model, and A method for video decoding characterized by including deriving the chroma residual scaling factor for decoding the CU based on the segment index. Claim 4 In paragraph 1, the plurality of reconstructed luma samples are, Generating luma prediction samples and luma residual samples, and Adding the above-mentioned luminance residual samples to the above-mentioned luminance prediction samples, obtaining the added luminance samples, and, A method for video decoding characterized by performing clipping the above-mentioned added luma samples into a dynamic range of coding bit depth. Claim 5 A method for video decoding according to claim 4, characterized in that the bit depth is 14 bits. Claim 6 A method for video decoding according to claim 1, characterized in that the plurality of reconstructed luma samples are derived based on forward-mapped inter-predicted samples. Claim 7 In claim 1, the predetermined region is a 64x64 region where the CU is located, and the region adjacent to the predetermined region is, A 1x64 area immediately above the above-mentioned predetermined area, and A method for video decoding characterized by including a 64x1 area immediately to the left of the above-mentioned predetermined area. Claim 8 As a method for video encoding, the above method is, Obtaining the average of a plurality of reconstructed luminance samples adjacent to a predetermined region where the above coding unit (CU) is located; Deriving a chroma residual scaling factor by directly using the average of the plurality of reconstructed lumina samples without any clipping; and A method for video encoding characterized by including transmitting a bitstream including the above-mentioned coding unit (CU). Claim 9 A method for video encoding according to claim 8, characterized in that the average of the plurality of reconstructed luma samples is the arithmetic mean of the plurality of reconstructed luma samples. Claim 10 In paragraph 8, deriving the chroma residual scaling factor by directly using the average of the plurality of reconstructed luminance samples without any clipping is, Identifying the segment index for the above mean in a pre-defined segment-by-segment linear model, and A method for video encoding characterized by including deriving the chrome residual scaling factor based on the above segment index. Claim 11 In paragraph 8, the plurality of reconstructed luma samples are, Generating luminance prediction samples and luminance residual samples, and Adding the above luminance residual samples to the above luminance prediction samples, and A method for video encoding characterized by performing clipping the above-mentioned added luma samples into a dynamic range of coding bit depth. Claim 12 A method for video encoding according to claim 11, characterized in that the bit depth is 14 bits. Claim 13 A method for video encoding according to claim 8, characterized in that the plurality of reconstructed luma samples are derived based on forward-mapped inter-predicted samples. Claim 14 In paragraph 8, the predetermined region is a 64x64 region where the CU is located, and the region adjacent to the predetermined region is, A 1x64 area immediately above the above-mentioned predetermined area, and A method for video encoding characterized by including a 64x1 area immediately to the left of the above-mentioned predetermined area. Claim 15 As a computing device, the computing device is, One or more processing units; and A computing device comprising a memory coupled to one or more processing units, wherein the electronic device is configured to perform the method of any one of claims 1 to 14. Claim 16 A non-transient computer-readable storage medium for storing a bitstream generated by a method for video encoding of any of claims 8 through 14. Claim 17 As a method for storing a bitstream, the method is, Performing a method for video encoding of any of claims 8 through 14 to generate a bitstream; and A method for storing a bitstream characterized by including storing the bitstream.

Images