VIDEO ENCODING METHOD, COMPUTING DEVICE, NON-TRANSITORY COMPUTER-READABLE STORAGE MEDIUM, AND METHOD FOR STORING BITSTREAMS - Patent application

Abstract

To provide a video coding method, a computing device, a non-transitory computer-readable storage medium and a method of storing a bitstream that reduce additional requirement of on-chip memory and computational complexity.SOLUTION: A video coding method comprises: acquiring an average of a plurality of reconstructed luma samples in a first predetermined region neighboring to a second predetermined region where a coding unit is located; and directly using the average of the plurality of reconstructed luma samples, without any clipping, in deriving a chroma residual scaling factor; and forming a bitstream by a luma mapping with chroma scaling related information.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] TECHNICAL FIELD This disclosure relates generally to video coding. [Background technology]

[0002] This section provides background information related to the present disclosure. Information contained in this section should not necessarily be construed as prior art.

[0003] Any of a variety of video encoding techniques may be used to compress the video data. The video encoding may be performed according to one or more video encoding standards. Some exemplary video encoding standards include versatile video coding (VVC), joint exploration test model (JEM) coding, high-efficiency video coding (H.265 / HEVC), advanced video coding (H.264 / AVC), and moving picture experts group (MPEG) coding.

[0004] Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit the redundancy inherent in a video image or sequence. One goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation of video quality.

[0005] The first version of the HEVC standard was completed in October 2013 and provides approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard H.264 / MPEG AVC. While the HEVC standard offers significant coding improvements over its predecessor, there is evidence that better coding efficiency can be achieved with additional coding tools than HEVC. Based on this, VCEG and MPEG have begun work to explore new coding techniques for future video coding standardization. The Joint Video Exploration Team (JVET) was formed by ITU-T VECG and ISO / IEC MPEG in October 2015 to begin significant research into advanced technologies that could enable substantial improvements in coding efficiency. One reference software, called the joint exploration model (JEM), is: It was maintained by JVET by integrating some additional coding tools on top of the HEVC Test Model (HM).

[0006] In October 2017, ITU-T and ISO / IEC issued a joint call for proposals (CfP) for video compression with capabilities exceeding HEVC. In April 2018, the 10th JVET meeting received and evaluated 23 CfP responses, demonstrating an approximately 40% improvement in compression efficiency over HEVC. Based on these evaluation results, a new project was launched to develop a new generation video coding standard named Versatile Video Coding (VVC). In the same month, a piece of reference software called the VVC Test Model (VTM) was established to demonstrate a reference implementation of the VVC standard.

[0007] Prediction methods utilized in video coding typically involve performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video data, and are typically associated with block-based video coding. Like HEVC, VVC is built on a block-based hybrid video coding framework.

[0008] In block-based video coding, the input video signal is processed block by block. For each block (also known as a coding unit (CU)), spatial prediction and / or temporal prediction may be performed. In newer video coding standards, such as the current VVC design, blocks may be further divided based on multi-type tree structures, including not only quadtrees but also binary and / or ternary trees. This allows for better adjustment of various local characteristics.

[0009] Spatial prediction (also known as "intra prediction") uses pixels from samples of already coded neighboring blocks (called reference samples) in the same video picture / slice to predict the current block. Spatial prediction reduces the spatial redundancy inherent in video signals.

[0010] During the decoding process, the video bitstream is first entropy decoded in an entropy decoding unit. The coding mode and prediction information are sent to either a spatial prediction unit (when intra-coded) or a temporal prediction unit (when inter-coded) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit and an inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then summed. The reconstructed block may further undergo in-loop filtering before being stored in a reference picture store. The reconstructed video in the reference picture store is then sent to drive a display device and used to predict future video blocks.

[0011] In newer video coding standards, such as the current VVC design, the coding tool luma mapping with chroma scaling (LMCS) can be applied before in-loop filtering. LMCS aims to adjust the dynamic range of the input signal to improve coding efficiency. Summary of the Invention [Problem to be solved by the invention]

[0012] However, the current design of LMCS uses different domain mappings in various decoding modules, which incurs computational complexity and additional on-chip memory requirements. Furthermore, the current design of LMCS uses different luma prediction sample values ​​to derive luma and chroma scaling factors, which introduces additional complexity. Furthermore, the current design of LMCS requires postponing the reconstruction of chroma residual samples until after the successful completion of luma prediction samples, which increases the latency of the reconstruction of chroma residual samples, which is difficult to achieve with complex inter-prediction techniques such as Decoder-side Motion Vector Derivation (DMVR), Bi-Directional Optical Flow (BDOF), and Combined Inter and Intra Prediction (CIIP). This requires the successful completion of sequential application of the mode coding tools. The current design of LMCS also performs unnecessary clipping operations during the chroma residual scaling factor derivation process, further increasing computational complexity and requiring additional on-chip memory. [Means for solving the problem]

[0013] This section provides a general overview of the disclosure, but is not an exhaustive disclosure of its entire scope or all of its features.

[0014] According to a first aspect of the present application, a plurality of reconstructed luma samples from a first predetermined area adjacent to a second predetermined area in which the CU is located are selected during decoding of the CU, an average of the plurality of reconstructed luma samples is calculated, and the average of the plurality of reconstructed luma samples is directly used without clipping when deriving a chroma residual scaling factor for decoding the CU.

[0015] According to a second aspect of the present application, a computing device includes one or more processors, a memory, and a plurality of programs stored in the memory, the programs, when executed by the one or more processors, causing the computing device to perform the operations described above in the first aspect of the present application.

[0016] According to a third aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by a computing device having one or more processors, the programs, when executed by the one or more processors, causing the computing device to perform the operations described above in the first aspect of the present application. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram illustrating an example block-based hybrid video encoder that can be used in many video coding standards. [Figure 2] FIG. 1 is a block diagram illustrating an example video decoder that can be used in many video coding standards. [Figure 3] FIG. 1 is a diagram of block partitioning in a multi-type tree structure that may be used in many video coding standards. [Figure 4] 10 is a flowchart showing a decoding process when LMCS is applied. [Figure 5] FIG. 1 is an explanatory diagram of a BDOF process. [Figure 6] 10 is a flowchart illustrating the workflow of chroma residual scaling in LMCS when DMVR, BDOF, and CIIP are all enabled. [Figure 7] 1 is a flowchart showing the steps of the prediction dependent residual scaling (PDRS) procedure. [Figure 8] 10 is a flowchart showing a workflow of a decryption process when a PDRS procedure is applied in LMCS processing. [Figure 9] FIG. 1 is an illustration of the residual mapping error caused by simply using prediction samples to derive scaling factors. [Figure 10] 10 is a flowchart illustrating steps in a chroma sample reconstruction procedure. [Figure 11] 10 is a flow chart illustrating steps of a second chroma sample reconstruction procedure. [Figure 12] 10 is a flowchart illustrating the workflow of the LMCS decoding process in an example of a second chroma sample reconstruction procedure in which DMVR, BDOF, and CIIP are not applied to generate luma prediction samples for chroma scaling. [Figure 13] 10 is a flowchart illustrating a workflow of an LMCS decoding process in a second example of a second chroma sample reconstruction procedure in which an initial uni-prediction signal is applied to generate luma prediction samples for chroma scaling. [Figure 14] 10 is a flowchart illustrating steps in a chroma residual sample reconstruction procedure. [Figure 15] 10 is a flowchart illustrating a workflow of an LMCS decoding process in one or more embodiments of a chroma residual sample reconstruction procedure. [Figure 16] 10 is a flowchart illustrating the workflow of the LMCS decoding process in one alternative embodiment of the chroma residual sample reconstruction procedure. [Figure 17] 10 is a flowchart illustrating steps of a second chroma residual sample reconstruction procedure. [Figure 18] 1 is a flowchart showing the steps of a first aspect of the present disclosure. [Figure 19] FIG. 1 is a diagram of the areas involved in an embodiment of the first aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0018] A set of exemplary, non-limiting embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. Variations in structure, method, or function may be implemented by those skilled in the art based on the examples presented herein, and all such variations are within the scope of the present disclosure. Where no contradiction exists, the teachings of different embodiments may, but need not, be combined with each other. do not have.

[0019] The terms used in this disclosure are not intended to limit the disclosure, but to describe specific examples. As used in this disclosure and the appended claims, the singular forms "a," "an," and the like also refer to the plural, unless the context clearly dictates otherwise. It should be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0020] Terms such as "first," "second," and "third" may be used herein to describe various pieces of information, but it should be understood that the information should not be limited by these terms. These terms are used only to distinguish one category of information from another category of information. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information, without departing from the scope of this disclosure. As used herein, the terms "if," "if," and the like may be understood to mean "when," "in," or "in response to," etc., depending on the context.

[0021] Throughout this specification, the singular or plural references to "an embodiment," "another embodiment," etc. mean that one or more particular features, configurations, etc. described in connection with an embodiment are included in at least one embodiment of the present disclosure. Thus, appearances of the phrases "in one embodiment," "in an embodiment," "in another embodiment," etc. throughout this specification do not necessarily all refer to the same embodiment. Furthermore, particular features, structures, or characteristics in one or more embodiments may be combined in any suitable manner.

[0022] Conceptually, many video coding standards, including those discussed above in the Background section, are similar: for example, virtually all video coding standards use block-based processing and share similar video coding block diagrams to achieve video compression.

[0023] Figure 1 shows a block diagram of an exemplary block-based hybrid video encoder 100 that may be used in many video coding standards. In encoder 100, a video frame is divided into multiple video blocks for processing. For each given video block, a prediction is formed based on either inter-prediction or intra-prediction techniques. In inter-prediction, one or more predictors are formed based on pixels from a previously reconstructed frame through motion estimation and motion compensation. In intra-prediction, a predictor is formed based on reconstructed pixels of a current intra-frame. Through mode decision, the best predictor may be selected to predict the current block.

[0024] A prediction residual, which represents the difference between the current video block and its predictor, is sent to a transform circuit 102. The transform coefficients are then sent from the transform circuit 102 to a quantization circuit 104 for entropy reduction. The quantized coefficients are then provided to an entropy coding circuit 106 to generate a compressed video bitstream. As shown in FIG. 1, prediction-related information 110 from an inter-prediction circuit and / or an intra-prediction circuit 112, such as video block partition information, motion vectors, reference picture indices, and intra-prediction modes, is also provided via the entropy coding circuit 106 and stored in the compressed video bitstream 114.

[0025] The encoder 100 also requires decoder-related circuitry to reconstruct pixels for prediction. First, a prediction residual is reconstructed via an inverse quantization circuit 116 and an inverse transform circuit 118. This reconstructed prediction residual is combined with a block predictor 120 to generate a prediction for the current video. Generate unfiltered reconstructed pixels for the ob-block.

[0026] Temporal prediction (also called "inter-prediction" or "motion-compensated prediction") uses reconstructed pixels from an already-encoded video picture to predict a current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. Additionally, if multiple reference pictures are supported, one reference picture index is additionally sent, which is used to identify which reference picture in the reference picture store the temporal prediction signal comes from.

[0027] After spatial and / or temporal prediction is performed, an intra / inter mode decision circuit 121 in encoder 100 selects the best prediction mode, for example, based on a rate-distortion optimization method. Block predictor 120 is then subtracted from the current video block, and the resulting prediction residual is decorrelated using transform circuit 102 and quantization circuit 104. The resulting quantized residual coefficients are inverse quantized by inverse quantization circuit 116 and inverse transformed by inverse transform circuit 118 to form a reconstructed residual, which is then added back to the prediction block to form the reconstructed signal for the CU. Other techniques, such as a deblocking filter, sample adaptive offset (SAO), and / or adaptive in-loop filter (ALF), may also be used. An additional in-loop filter 115 may be applied to the reconstructed CU before it is placed in a reference picture store in a picture buffer 117 and used to encode future video blocks. To form the output video bitstream 114, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit 106 where they are further compressed and packed to form the bitstream.

[0028] For example, deblocking filters are available in AVC, HEVC, as well as the current version of VVC. HEVC defines an additional in-loop filter called sample adaptive offset (SAO) to further improve coding efficiency. In the current VVC standard, yet another in-loop filter called adaptive loop filter (ALF) is under active consideration and is likely to be included in the final standard.

[0029] These in-loop filter operations are optional. Performing these operations helps improve coding efficiency and visual quality. They can also be turned off as a decision is made by the encoder 100 to save computational complexity.

[0030] Note that intra prediction is typically based on unfiltered reconstructed pixels, while inter prediction is based on filtered reconstructed pixels if these filter options are turned on by the encoder 100.

[0031] Figure 2 is a block diagram illustrating an exemplary video decoder 200 that may be used in many video coding standards. This decoder 200 is similar to the reconstruction-related section present in the encoder 100 of Figure 1. In the decoder 200 (Figure 2), an input video bitstream 201 is first decoded through entropy decoding 202 to derive quantized coefficient levels and prediction-related information. The quantized coefficient levels are then processed through inverse quantization 204 and an inverse transform 206 to obtain a reconstructed prediction residual. A block prediction mechanism implemented in intra / inter mode selection 212 is configured to perform either intra prediction 208 or motion compensation 210 based on the decoded prediction information. The set of unfiltered reconstructed pixels is then reconstructed from the inverse transform 206. The resulting prediction residual is obtained by summing, using adder 214, the prediction output generated by the block predictor.

[0032] The reconstructed blocks may further pass through an in-loop filter 209 before being stored in a picture buffer 213, which serves as a reference picture store. The reconstructed video in the picture buffer 213 may then be sent to drive a display device, as well as used to predict future video blocks. In situations where the in-loop filter 209 is turned on, a filtering operation is performed on these reconstructed pixels to derive the final reconstructed video output 222.

[0033] In video coding standards such as HEVC, blocks can be divided based on a quadtree. Newer video coding standards, such as the current VVC, adopt more partitioning methods, and coding tree units (CTUs) can be divided into CUs based on quadtrees, binary trees, or ternary trees to adapt to changing local characteristics. The separation of CUs, prediction units (PUs), and transform units (TUs) does not exist in most coding modes in the current VVC, and each CU is always used as the basic unit for both prediction and transformation without further partitioning. However, in some specific coding modes, such as the sub-partition coding mode, each CU may still contain multiple TUs. In a multi-type tree structure, one CTU is first partitioned using a quadtree structure. Then, each quadtree leaf node can be further partitioned using a binary tree structure and a ternary tree structure.

[0034] 3 shows five partition types used in current VVC: 4-partition 301, horizontal 2-partition 302, vertical 2-partition 303, horizontal 3-partition 304, and vertical 3-partition 305. In a situation where a multi-type tree structure is used, one CTU is first partitioned by a quadtree structure. Then, each quadtree leaf node can be further partitioned by a binary tree structure and a ternary tree structure.

[0035] Using one or more of the example block partitions 301, 302, 303, 304, or 305 of Figure 3, spatial prediction and / or temporal prediction may be performed using the configuration shown in Figure 1. Spatial prediction (or "intra-prediction") uses pixels from samples of already-encoded neighboring blocks (called reference samples) in the same video picture / slice to predict a current video block. Spatial prediction reduces spatial redundancy inherent in video signals.

[0036] Newer video coding standards, such as VVC, now add a new coding tool called Luma Mapping with Chroma Scaling (LMCS), which is applied before the loop filters (e.g., deblocking filter, SAO, and ALF).

[0037] In general, LMCS has two main modules: the first is an in-loop mapping of the luma component based on an adaptive piecewise linear model, and the second is a luma-dependent chroma residual scaling.

[0038] Figure 4 shows a modified decoding process where LMCS is applied. In Figure 4, several blocks represent the decoding module, which is performed in the mapped domain, including entropy decoding 401, inverse quantization 402, inverse transform 403, luma intra prediction 404, and luma sample reconstruction 405 (i.e., summing luma prediction samples Y'pred and luma residual samples Y'res to generate reconstructed luma samples Y'recon). Several other blocks represent motion compensated prediction 409, chroma intra prediction 412, and chroma sample reconstruction 413 (i.e., adding luma prediction samples Y'pred and luma residual samples Y'res to generate reconstructed luma samples Y'recon). 4 shows decoding modules performed in the original (i.e., unmapped) domain, including forward mapping 410 and backward (or inverse) mapping 406 of luma samples, and chroma residual scaling 411. Also shown in FIG. 4 are all reference pictures in the original domain, stored in the decoded picture buffers (DPBs) 408 (for luma) and 415 (for chroma).

[0039] The in-loop mapping of LMCS aims to adjust the dynamic range of the input signal to improve coding efficiency. The in-loop mapping of luma samples in existing LMCS designs is constructed based on two mapping functions: one forward mapping function FwdMap and one corresponding inverse mapping function InvMap. The forward mapping function is signaled from the encoder to the decoder using a piecewise linear model with 16 equally sized pieces. The inverse mapping function can be derived directly from the forward mapping function and therefore does not need to be signaled.

[0040] The parameters of the luma mapping model are signaled at the slice level. A presence flag is first signaled to indicate whether a luma mapping model should be signaled for the current slice. If a luma mapping model exists in the current slice, corresponding piecewise linear model parameters are further signaled. Based on the piecewise linear model, the dynamic range of the input signal is divided into 16 segments of equal size in the original domain, and each segment is mapped to a corresponding segment. For a given segment in the original domain, its corresponding segment in the mapped domain may have the same or different size. The size of each segment in the mapped domain is indicated by the number of codewords (i.e., mapped sample values) of that segment. For each segment in the original domain, linear mapping parameters can be derived based on the number of codewords in the corresponding segment in the mapped domain. For example, when the input is at 10-bit depth, if each of the 16 segments in the original domain has 64 pixel values ​​and each of the segments in the mapped domain also has 64 codewords assigned to it, it indicates a simple one-to-one mapping (i.e., a mapping in which each sample value is unchanged). The signaled number of codewords for each segment in the mapped domain is used to calculate a scaling factor and adjust the mapping function accordingly for that segment. Additionally, at the slice level, another LMCS control flag is signaled to enable / disable LMCS for the slice.

[0041] For each segment, the corresponding piecewise linear model is defined as described in the box immediately following this paragraph:

number

[0042] As shown in FIG. 4, two different domains need to be operated on during the LMCS process. For each CU coded via an inter-prediction mode ("inter CU"), its motion-compensated prediction is performed in the original domain. However, because the reconstruction of the luma component (i.e., the addition of luma prediction samples and luma residual samples) is performed in the mapped domain, the motion-compensated luma prediction Ypred needs to be mapped from the original domain to a value Y'pred in the mapped domain via a forward mapping function 410, i.e., Y'pred=FwdMap(Ypred), and then Y'pred is used for pixel reconstruction 405. On the other hand, for each CU coded via an intra-prediction mode ("intra CU"), after intra prediction 404 is performed in the mapped domain (as shown in FIG. 4), Y'pred is used for pixel reconstruction 405, so no mapping of prediction samples is required. Finally, after generating the reconstructed luma samples Y'recon, an inverse mapping function 406 is applied to convert the reconstructed luma samples Y'recon to Yrecon in the original domain, and then proceed to the luma DPB 408, i.e., Yrecon=InvMap(Y'recon). Unlike the forward mapping of predicted samples 410, which only needs to be applied to inter-CUs, the inverse mapping of reconstructed samples 406 needs to be applied to both inter-CUs and intra-CUs.

[0043] In summary, on the decoder side, an in-loop luma mapping of the current LMCS is performed such that the luma predicted samples Ypred are first transformed, if necessary, into the mapped domain: Y'pred = FwdMap(Ypred). The mapped predicted samples are then summed with the decoded luma residuals to form reconstructed luma samples in the mapped domain: Y'recon = Y'pred + Y'res. Finally, an inverse mapping is applied to transform the reconstructed luma samples Y'recon back to the original domain: Yrecon = InvMap(Y'recon). On the encoder side, the luma residuals are coded in the mapped domain, so they are generated as the difference between the mapped original luma samples and the mapped luma predicted samples: Y'res = FwdMap(Y'recon). p(Yorg)-FwdMap(Ypred).

[0044] The second step of LMCS, luma-dependent chroma residual scaling, is designed to compensate for the quantization precision interaction between a luma signal and its corresponding chroma signal when in-loop mapping is applied to the luma signal. Whether chroma residual scaling is enabled or disabled is also signaled in the slice header. When luma mapping is enabled and binary tree partitioning of the luma and chroma components is disabled for the current slice, an additional flag is signaled to indicate whether luma-dependent chroma residual scaling is applied. When luma mapping is not used or binary tree partitioning is enabled for the current slice, luma-dependent chroma residual scaling is always disabled. Furthermore, chroma residual scaling is always disabled for CUs containing four or fewer chroma samples.

[0045] For both intra and inter CUs, the scaling parameters used to scale the chroma residual depend on the average of the corresponding mapped luma prediction samples. The scaling parameters are derived as described in the box immediately following this paragraph:

number

[0046] 4 also shows the computation of the average of luma prediction samples for luma-dependent chroma residual scaling. For inter CUs, the forward-mapped luma prediction Y'pred is fed to chroma residual scaling 411 along with the scaled chroma residual CresScale to derive the chroma residual Cres, which is fed to chroma reconstruction 413 along with the chroma prediction Cpred to derive the reconstructed chroma value Crecon. For intra CUs, intra prediction 404 generates Y'pred, which is already in the mapped domain, and is fed to chroma residual scaling 411 as in the inter CU case.

[0047] Unlike luma mapping, which is performed on a sample-by-sample basis, CScaleInv scales chroma Fixed for the entire CU. If CScaleInv is given, chroma residual scaling is applied as described in the box immediately following this paragraph.

number

[0048] New video coding standards such as VVC have introduced new coding tools, some of which are BDOF (Bi-Directional Optical Flow), DMV (Digital Multimedia Video), and so on. These are R (Decoder-side Motion Vector Refinement), CIIP (Combined Inter and Intra Prediction), Affine Mode, and PROF (Prediction Refinement with Optical Flow) for Affine Mode.

[0049] In the current VVC, BDOF (bi-directional optical flow) is used for bidirectional predictive coding. It is applied to refine the predicted samples of the lock.

[0050] 5 is an illustration of the BDOF process. BDOF is a sample-by-sample motion refinement performed in block-based motion compensation prediction when bidirectional prediction is used. The motion refinement (vx,vy) of each 4x4 sub-block 501 is calculated by minimizing the difference between the predicted samples 502 and 503 of reference picture list 0 (L0) and reference picture list 1 (L1) after BDOF has been applied within a 6x6 window Ω around the sub-block.

[0051] Specifically, the motion refinement values ​​(vx, vy) are derived as described in the box immediately following this paragraph.

number

[0052] The values ​​S1, S2, S3, S5, and S6 in the previous box are written in the box immediately following this paragraph. Further calculations are performed as described in the accompanying drawings.

number

[0053]

number

number

[0054] Based on the motion refinement derived as described in the box immediately following paragraph

[0072] above, the final bi-prediction samples for the CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the optical flow model, as shown in the box immediately following this paragraph.

[0055] According to the above-mentioned bit depth control method, the maximum bit depth of the intermediate parameters in the entire BDOF process does not exceed 32 bits, and the maximum input to the multiplier is within 15 bits, that is, That is, it is guaranteed that one 15-bit multiplier is sufficient for the BDOF implementation.

number

[0056] DMVR is a bi-prediction technique used for merging blocks with two initially signaled MVs, which can be further refined by using bilateral matching prediction.

[0057] Specifically, DMVR uses bilateral matching to derive the motion information of the current CU by finding the best match between two blocks along the motion trajectory of the current CU in two different reference pictures. The cost function used in the matching process is the low-subsampled sum of absolute differences (SAD). After the matching process is performed, the refined MV is used for motion compensation in the prediction stage, i.e., temporal motion vector prediction for subsequent pictures, and the unrefined MV is used for motion vector prediction between the motion vector of the current CU and its spatially adjacent motion vectors.

[0058] Under the assumption of continuous motion trajectories, the motion vectors MV0 and MV1 pointing to two reference blocks are proportional to the temporal distances between the current picture and the two reference pictures, i.e., TD0 and TD1. As a special case, when the current picture is temporally between two reference pictures and the temporal distances from the current picture to the two reference pictures are the same, the bilateral matching becomes mirror-based bidirectional MV.

[0059] In current VVC, inter-prediction and intra-prediction methods are used in a hybrid video coding scheme, where each PU is only allowed to select inter-prediction or intra-prediction to exploit correlation in either the temporal or spatial domain, but never both. However, as pointed out in previous literature, the residual signals generated by inter-predicted and intra-predicted blocks may exhibit very different characteristics. Therefore, if the two types of prediction can be efficiently combined, we can expect a more accurate prediction to reduce the energy of the prediction residual and improve coding efficiency. In addition, in video content, the movement of moving objects can be complex. For example, there may be areas that contain both old content (e.g., objects included in previously coded pictures) and emerging new content (e.g., objects different from the previously coded pictures). In such scenarios, neither inter-prediction nor intra-prediction can provide an accurate prediction of the current block.

[0060] To further improve prediction efficiency, the VVC standard adopts combined inter and intra prediction (CIIP), which combines inter prediction and intra prediction for one CU coded in merge mode. Specifically, for each merge CU, one additional flag is signaled to indicate whether CIIP is enabled for the current CU. When the flag is equal to 1, CIIP applies only planar mode to generate intra-predicted samples of luma and chroma components. Furthermore, equal weights (i.e., 0.5) are applied to average the inter-predicted samples and intra-predicted samples as the final predicted samples of the CIIP CU.

[0061] VVC also supports affine modes for motion-compensated prediction. In HEVC, only a translational motion model is applied for motion-compensated prediction. In the real world, there are many types of motion, such as zoom-in / zoom-out, rotation, perspective motion, and other irregular motion. In VVC, affine motion-compensated prediction is applied by signaling one flag for each inter-coded block to indicate whether affine motion compensation prediction is applied to translational motion or whether an affine motion model is applied to inter prediction. In the current VVC design, two affine modes are supported for one affine-coded block, including a four-parameter affine mode and a six-parameter affine mode.

[0062] The four-parameter affine model has the following parameters: two parameters for horizontal and vertical translational motion, one parameter for zoom motion, and one parameter for rotational motion in both directions. The horizontal zoom parameter is the same as the vertical zoom parameter. The horizontal rotation parameter is equal to the vertical rotation parameter. To achieve better adaptation of the motion vectors and affine parameters, VVC converts these affine parameters into two MVs (also called control point motion vectors (CPMVs)) located at the upper left and upper right corners of the current block. The affine motion field of a block is described by two control point MVs (V0, V1).

[0063] Based on the movement of the control points, the motion field (vx, vy) of one affine coded block is calculated as described in the box immediately following this paragraph.

number

[0064] The 6-parameter affine mode has two parameters for horizontal and vertical translational motion, one parameter for zoom motion, and one parameter for horizontal rotational motion, one parameter for zoom motion, and one parameter for vertical rotational motion. The 6-parameter affine motion model is coded with three MVs in three CPMVs.

[0065] The three control points of a six-parameter affine block are located in the top-left, top-right, and bottom-left corners of the block. Movement at the top-left control point is associated with translational movement, movement at the top-right control point is associated with horizontal rotation and zoom movement, and movement at the bottom-left control point is associated with vertical rotation and zoom movement. Compared to the four-parameter affine motion model, the horizontal rotation and zoom movement of the six-parameter affine block may not be the same as their vertical counterparts.

[0066] Assuming (V0, V1, V2) are the MVs of the top-left, top-right, and bottom-left corners of the current block, the motion vectors (vx, vy) of each sub-block are derived using the three MVs at the control points as described in the box immediately following this paragraph.

number

[0067] To improve the accuracy of affine motion compensation, prediction refinement using PROF (Prediction Refinement with Optical Flow) is currently being considered in the current VVC, which refines subblock-based affine motion compensation based on an optical flow model. Specifically, after performing sub-block-based affine motion compensation, the luma prediction samples of one affine block are modified by one sample refinement value derived based on the optical flow formula. In detail, the operation of PROF can be summarized as the following four steps:

[0068] In step 1, sub-block based affine motion compensation is performed to generate sub-block predictions I(i,j) using the sub-block MVs as derived in the box immediately following paragraph

[0084] above for the 4-parameter affine model and in the box immediately following paragraph

[0087] above for the 6-parameter affine model.

[0069] In step 2, the spatial gradients gx(i,j) and gy(i,j) of each prediction sample are calculated as described in the box immediately following this paragraph.

number

[0070] Furthermore, in step 2, one additional row / column of prediction samples needs to be generated on each side of one sub-block to calculate the gradient. To reduce memory bandwidth and complexity, samples on the extension boundary are copied from the nearest integer pixel position in the reference picture to avoid an additional interpolation process.

[0071] In step 3, the luminance prediction refinement value is calculated as described in the box immediately following this paragraph.

number

[0072] Furthermore, in the current PROF design, after adding the prediction refinement to the original prediction samples, As a fourth step, a clipping operation is performed to clip the values ​​of the refined prediction samples to within 15 bits, as described in the box immediately following this paragraph.

number

[0073] Since the affine model parameters and pixel positions relative to the sub-block center do not change from sub-block to sub-block, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks within the same CU. Let Δx and Δy be the horizontal and vertical offsets from sample position (i,j) to the center of the sub-block to which the sample belongs, and Δv(i,j) can be derived as described in the box immediately following this paragraph.

number

[0074] Based on the affine sub-block MV derivation equations in the boxes immediately following paragraphs

[0084] and

[0087] above, the MV difference Δv(i,j) can be derived as described in the box immediately following this paragraph.

number

[0075] According to the current LMCS design, chroma residual samples are scaled based on their corresponding luma prediction samples. When newer coding tools are enabled for an inter CU, the luma prediction samples used to scale chroma residual samples through LMCS in this inter CU are obtained at the end of the sequential application of these newer coding tools.

[0076] 6 is a flowchart illustrating the workflow of chroma residual scaling in LMCS when DMVR, BDOF, and CIIP are all enabled. The outputs from luma L0 prediction 601 and L1 prediction 602 are sequentially fed to DMVR 603 and BDOF 604, and the resulting luma inter prediction 621, along with luma intra prediction 622, are fed to luma intra prediction 605 through averaging 606 to generate averaged luma prediction 623, which, along with chroma residual 608, is fed to chroma residual scaling 607, where chroma residual scaling 607, chroma prediction 610, and chroma reconstruction 609 can work together to generate the final result.

[0077] Current LMCS designs present three challenges to the video decoding process. First, mapping between different domains requires additional computational complexity and on-chip memory. Second, the fact that luma and chroma scaling factor derivation uses different luma predictors introduces additional complexity. Third, the interaction between LMCS and newer coding tools introduces latency into the decoding process.

[0078] First, in the current LMCS design, both the reconstructed samples in the original domain and the mapped domain are used in various decoding modules. These samples often need to be transformed from one domain to another between different decoding modules, which can lead to both higher computational complexity and more on-chip memory.

[0079] Specifically, for intra, CIIP, and IBC modes, mapped domain reference samples from neighboring reconstructed domains of one current CU are used to generate predicted samples. However, for inter modes, motion compensation prediction is performed using original domain reconstructed samples of temporal reference pictures as references. The reconstructed samples stored in the DPB are also in the original domain.

[0080] For example, for inter CUs, because the luma reconstruction operation (i.e., the addition of predicted samples and residual samples) is performed in the mapped domain, inter predicted luma samples generated in the original domain need to be transformed to the mapped domain before being used for luma sample reconstruction. In another example, for both intra CUs and inter CUs, inverse (or backward) mapping is always applied to transform reconstructed luma samples from the mapped domain to the original domain before storing them in the DPB. Such a design not only increases computational complexity due to the additional forward / backward mapping operation, but also requires more on-chip memory to maintain multiple versions of the reconstructed samples.

[0081] In an actual hardware implementation, the forward and backward (or inverse) mapping functions FwdMap and InvMap may be implemented either using a look-up table (LUT) or calculated on the fly. When a LUT-based solution is used, the possible output elements from the functions FwdMap, InvMap, and cScaleInv may be pre-calculated and pre-stored as a LUT, which may then be used for the luma mapping and chroma residual scaling operations of all CUs in the current slice. Assuming the input video is 10-bit, each of the LUTs for FwdMap and InvMap may contain 2 10= 1024 elements, and each element of the LUT is 10 bits. Therefore, the total storage for the LUTs of the forward and reverse luma mapping is equal to 2 * 1024 * 10 = 20480 bits = 2560 bytes. Meanwhile, to derive the chroma scaling parameters CScaleInv, one 16-entry LUT table cScaleInv needs to be maintained in the encoder and decoder, with each chroma scaling parameter stored in 32 bits. Correspondingly, the memory size used to store the LUT cScaleInv is equal to 16 * 32 = 512 bits = 64 bytes. The difference between 2560 and 64 indicates the scale of the additional on-chip memory required by the forward and reverse (inverse) mapping operations.

[0082] Furthermore, in newer video coding standards, such as the current VVC, both intra prediction and deblocking filters use reconstructed samples of the neighboring blocks. Therefore, one additional row of reconstructed samples within the width of the current picture / slice needs to be maintained in a buffer, also known as a "line buffer" in video coding. The reconstructed samples in the line buffer are used as references for intra prediction and deblocking operations of at least the CU located in the first row in one CTU. According to existing LMCS designs, the intra prediction and deblocking filters use reconstructed samples in different domains. Therefore, additional on-chip memory is required to store both the original and mapped domain reconstructed samples, which nearly doubles the line buffer size.

[0083] One implementation choice to avoid doubling the line buffer size is to use domain mapping The solution is to perform the operation on the fly, however this would incur a non-negligible increase in computational complexity.

[0084] Therefore, current designs of LMCS require additional computational complexity and on-chip memory due to the required mapping between different domains.

[0085] Second, although both the luma scaling factor derivation method and the chroma scaling factor derivation method in the current design of LMCS use luma predicted sample values ​​to derive corresponding scaling factors, there are differences between their corresponding operations.

[0086] For luma residual scaling, the scaling factor is derived on a sample-by-sample basis by allowing each luma residual sample to have its own scaling factor. However, for chroma residual scaling, the scaling factor is fixed for the entire CU, i.e., all chroma residual samples within a CU share the same scaling factor, which is calculated based on the average of the mapped luma prediction samples.

[0087] Two different LUTs are used to calculate the scaling factors for the luma and chroma residuals: the input to the luma LUT is the mapping model segment index of the original luma prediction sampling value, while the input to the chroma LUT is the mapping model segment index of the average value of the mapped luma prediction sampling.

[0088] Such differences introduce additional complexity into the encoding process, and a harmonized approach to luma and chroma scaling factor derivation is desirable.

[0089] Third, for the chroma residual scaling in the current design of LMCS, newer coding tools, such as all three modules of DMVR, BDOF, and CIIP, can be invoked sequentially to generate luma prediction samples, which are then used to determine the scaling factor for the chroma residual. If the computational complexity of the three modules is large, waiting for their successful completion before performing the chroma residual scaling of LMCS may cause significant latency for the decoding of the chroma samples. For affine CUs, the PROF process may have latency issues because each affine CU may execute the PROF process followed by LMCS, which may cause latency issues for the decoding of the chroma samples.

[0090] Furthermore, in the current design of LMCS, unnecessary clipping operations are performed during the chroma residual scaling factor derivation process, further increasing the computational complexity and additional on-chip memory requirements.

[0091] This disclosure aims to solve or mitigate these challenges presented by current designs of LMCS, and more specifically, this disclosure discusses ways in which the complexity of LMCS for hardware codec implementations may be reduced while maintaining coding gain.

[0092] Instead of using the existing LMCS framework, which transforms the predicted / reconstructed samples via a mapping operation, we propose a new method, called prediction-dependent residual scaling (PDRS), to directly scale the predicted residual without sample mapping. The proposed method achieves similar effectiveness and coding efficiency as LMCS, but with much less implementation complexity.

[0093] In the PDRS procedure, a luma prediction sample is used to decode the luma residual samples, as shown in Figure 7. A sample is obtained (701), a scaling factor is derived using the luma prediction sample (702), the scaling factor is used to scale the luma residual sample (703), and a reconstructed luma sample is calculated by adding the luma prediction sample and the scaled luma residual sample (704).

[0094] Unlike existing LMCS methods that directly transform predicted / reconstructed luma samples into the mapped domain before calculating the luma prediction residual, in the proposed PDRS procedure, luma prediction residual samples are derived in the same way as the normal prediction process in the original domain without a mapping operation, followed by a scaling operation on the luma prediction residual. The scaling of the luma prediction residual depends on the corresponding luma prediction sample values ​​and the piecewise linear model. As a result, the forward and luma inverse mapping operations in current LMCS designs can be completely discarded, with all predicted and reconstructed samples included in the decoding process remaining in the original sample domain. Based on the above characteristics, we name the proposed method prediction-dependent residual scaling.

[0095] 8 is a flowchart showing the workflow of the decoding process when the PDRS procedure is applied in the LMCS process. This shows the elimination of the need for mapping between different domains. Here, except for the residual decoding module (e.g., entropy decoding 801, inverse quantization 802, and inverse transform 803), all other decoding modules (including intra- and inter-prediction 804, 809, 812, and 816, reconstruction 806 and 813, and all in-loop filters 807 and 814) operate in the original domain. Specifically, to reconstruct luma samples, the proposed approach in the PDRS procedure only needs to inversely scale luma prediction residual samples Yres to their original amplitude levels and then add them to luma prediction samples Ypred.

[0096] In the PDRS procedure, the forward and inverse luma sample mapping operations in existing LMCS designs are completely eliminated. This not only saves / reduces computational complexity, but also reduces potential storage size by saving LMCS parameters. For example, when a LUT-based solution is used to implement luma mapping, the storage previously used to store the two mapping LUTs FwdMap[] and InvMap[] (approximately 2560 bytes) in the proposed method is no longer required. Furthermore, unlike existing luma mapping methods that require storing reconstructed luma samples in both the original and mapped domains, the proposed method in the PDRS procedure generates and maintains all predicted and reconstructed samples only in the original domain. Correspondingly, compared to existing luma mapping, the proposed method in the PDRS procedure can effectively halve the line buffer size used to store reconstructed samples for intra prediction and deblocking.

[0097] According to one or more embodiments of the PDRS procedure, the luma prediction sample and the luma residual sample are from one and the same location in the luma prediction block and its associated residual block.

[0098] According to one or more embodiments of the PDRS procedure, deriving the scaling factors using the luma prediction samples includes dividing a range of possible luma prediction sample values ​​into a plurality of luma prediction sample segments, calculating one scaling factor for each of the plurality of luma prediction sample segments based on a predetermined piecewise linear model, and determining a scaling factor for the luma prediction sample based on the scaling factors of the plurality of luma prediction sample segments.

[0099] In one example, a luminance prediction is performed based on a scaling factor for a plurality of luminance prediction sample segments. Determining the scaling factor of the prediction sample includes allocating the luma prediction sample to one segment among a plurality of luma prediction sample segments, and calculating the scaling factor of the luma prediction sample as the scaling factor of the allocated luma prediction sample segment.

[0100] In this example, the plurality of luminance prediction sample segments includes 16 segments in a predetermined 16-section LUT table scaleForward, and the predetermined piecewise linear model for calculating one scaling factor for each of the plurality of luminance prediction sample segments includes 16 values ​​corresponding to the 16 segments in the predetermined LUT table scaleForward.

[0101] In the same example, the scaling factor is calculated based on the assigned luma prediction sample segment, as described in the box immediately following this paragraph.

number

[0102] In the same example, given a luminance scaling factor ScaleY, the luminance residual sample scaling method can be applied as described in the box immediately following this paragraph.

number

[0103] The motivation behind this example is that the forward mapping in current LMCS is based on a piecewise linear model: both the original luma samples and the luma predicted samples are in the same piecewise linear model. That is, when the original luma sample and the predicted luma sample are located in the same segment defined by two pivot points InputPivot[i] and InputPivot[i+1], the two forward mapping functions of the original luma sample and the predicted luma sample are exactly the same. Correspondingly, Y'res = FwdMap(Yorg) - FwdMap(Ypred) = FwdMap(Yorg - Ypred) == FwdMap(Yres). By applying inverse mapping to both sides of this equation, the corresponding decoder-side reconstruction operation can be expressed as Yrecon = Ypred + InvMap(Y'res).

[0104] In other words, in situations where both the original luma samples and luma prediction samples are located in the same partition, this can be achieved through one residual scaling operation in the decoding process, as the luma mapping method in LMCS is implemented in this possible implementation.

[0105] Although such a conclusion is derived based on the assumption that both the original luma sample and the luma predicted sample are located within the same segment defined by two pivot points InputPivot[i] and InputPivot[i+1], this possible implementation of this example can still be used as a simplification and / or approximation for the existing luma mapping operation in VVC even when the original luma sample and the luma predicted sample are located within different segments of the piecewise linear model. Experimental results show that such simplification and / or approximation has little impact on coding performance.

[0106] Again, this example is based on the assumption that both the original and predicted luma sample values ​​are located within the same segment in piecewise linear mode. In this case, the forward / backward mapping functions applied to the original and predicted luma samples are the same, and therefore it is safe to simply rely on the luma prediction samples to calculate the corresponding residual scaling factors.

[0107] However, when the predicted samples of a CU are not sufficiently accurate (e.g., in the case of an intra-predicted CU, where samples farther away from the reference sample are usually predicted less accurately), the predicted samples and the original samples often lie in different segments of the piecewise linear model. In this case, the scaling factors derived based on the predicted sample values ​​may not be reliable in reflecting the original mapping relationship between the residual samples in the original (i.e., unmapped) domain and the residual samples in the mapped domain.

[0108] FIG. 9 is an illustration of residual mapping errors caused by simply using predicted samples to derive scaling factors. In FIG. 9, the solid triangular points represent pivot control points of various segments in a piecewise linear function, the solid circular points represent original and predicted sample values, Yorg and Ypred are original and predicted samples in the original (i.e., unmapped) domain, and Y'org and Y'pred are mapped samples of Yorg and Ypred, respectively. Yres and Y'res are corresponding residuals in the original and mapped domains when an existing sample-based luma mapping method in VVC is applied, and Y'resScale is a mapped residual sample derived based on the proposed luma residual scaling method. As shown in FIG. 9, because the original and predicted samples are not in the same interval of the piecewise linear model, the scaling factor derived based on the predicted sample may not be accurate enough to generate a scaled residual (i.e., Y'resScale) that approximates the original residual in the mapped domain (i.e., Y'res).

[0109] In the second example, both the original and predicted luminance sample values ​​are in piecewise linear mode. There is no need to assume that the data are located within the same segment of the node.

[0110] In this second example, instead of deriving the scaling factor directly from the segment of the piecewise linear model in which the luminance prediction sample is located, the scaling factor is calculated as the average of the scaling factors of N adjacent segments (N is a positive integer).

[0111] In this second example, determining a scaling factor for a luminance prediction sample based on scaling factors of a plurality of luminance prediction sample segments includes allocating the luminance prediction sample to one of the plurality of luminance prediction sample segments and calculating the scaling factor for the luminance prediction sample as an average of scaling factors of several luminance prediction sample segments adjacent to the allocated luminance prediction sample segment.

[0112] More specifically, in one possible implementation of this second example, the scaling factor may be calculated based on the allocated luma prediction sample segments, as described in the following steps. 1) Find or get the segment index IdxY corresponding to the piecewise linear model to which PredY belongs in the original domain. 2) If Y'res ≥ 0, the luminance residual scaling factor is calculated as follows:

number

number

[0113] In a second possible implementation of this second example, which is otherwise identical to the implementation described above, the scaling factor may be calculated based on the assigned luma prediction sample segments, as described in the box immediately following this paragraph:

number

[0114] The above two possible implementations of this second example differ only in the selection of the N luminance prediction sample domain value segments based on the assigned segments.

[0115] In one chroma sample reconstruction procedure, as shown in FIG. 10 , a luma prediction sample value is obtained to decode both the luma residual sample and the chroma residual sample at the input position (1001), then a luma prediction sample associated with the luma residual sample is obtained (1002), then a chroma prediction sample associated with the chroma residual sample is obtained (1003), the luma prediction sample is used to derive a first scaling factor for the luma residual sample and a second scaling factor for the chroma residual sample (1004), the first scaling factor is used to scale the luma residual sample (1005), the second scaling factor is used to scale the chroma residual sample (1006), a reconstructed luma sample is calculated by adding the luma prediction sample and the scaled luma residual sample (1007), and a reconstructed chroma sample is calculated by adding the chroma prediction sample and the scaled chroma residual sample (1008).

[0116] The chroma sample reconstruction procedure aims to harmonize the scaling methods of the luma and chroma residuals to achieve a more unified design.

[0117] According to one or more embodiments of the chroma sample reconstruction procedure, the luma prediction sample value is the average of all luma prediction samples in the coding unit (CU) that contains the input location. In these embodiments, a chroma scaling derivation method is used to calculate a scaling factor for the luma residual; more specifically, instead of deriving one scaling factor separately for each luma residual sample, one shared scaling factor calculated based on the average of the luma prediction samples is used to scale the luma residual samples across the CU.

[0118] According to another embodiment of the chroma sample reconstruction procedure, the luma prediction sample value is the average of all luma prediction samples within a predefined sub-block subdivided from the coding unit (CU) containing the input location. In this embodiment, one CU is first equally divided into a number of M×N sub-blocks, and then, for each sub-block, all or part of the luma prediction samples are used to derive corresponding scaling factors that are used to scale both the luma and chroma residuals of the sub-block. Compared to the first method, the second method uses less correlated luma prediction samples outside the sub-block. Since the first method is excluded from calculating the scaling factor of the sub-block, the spatial accuracy of the estimated scaling factor can be improved. Meanwhile, the second method can also reduce the latency of the luma and chroma residual reconstruction, assuming that scaling of the luma and chroma residuals in one sub-block can be started immediately after the luma prediction of the sub-block is finished, i.e., without waiting for the complete generation of luma prediction samples for the entire CU.

[0119] According to a third embodiment of the chroma sample reconstruction procedure, the luma prediction sample domain values ​​include collocated luma prediction samples. In this embodiment, the luma residual scaling method is extended to scale the chroma residuals, and a different scaling factor for each chroma residual sample is derived based on its collocated luma prediction sample value.

[0120] In the above embodiment of the chroma sample reconstruction procedure, it is proposed to use the same LUT used to calculate the luma scaling factor for scaling the chroma residual. In one example, to derive the CU level scaling factor ScaleC for the chroma residual, one can do the following: 1) Calculate the average of the luminance prediction samples (represented in the original domain) within the CU, denoted as avgY. 2) Find or get the corresponding segment index IdxY of the piecewise linear model to which avgY belongs. 3) Calculate the value of ScaleC as follows: ScaleC=scaleForward[IdxY] where scaleForward[i], i=0...15 is a predefined 16-segment LUT, It is calculated as: scaleForward[i]=(OrgCW< <SCALE_FP_PREC) / SignaledCW[i] where OrgCW and SignaledCW[i] are the numbers of codewords of the ith segment in the original and mapped domains, and SCALE_FP is the precision of the scaling factor.

[0121] The above example can be easily extended if the scaling factor for the chroma residual is derived for each sub-block of the current CU. In that case, in the first step above, avgY is calculated as the average of the luma prediction samples in the original domain of the sub-block, while steps 2 and 3 remain the same.

[0122] In the second chroma sample reconstruction procedure, as shown in FIG. 11, during the luma prediction process for a coding unit (CU), multiple luma prediction samples are obtained by skipping some of the predetermined intermediate luma prediction stages (1101), the obtained multiple luma prediction samples are used to derive scaling factors for the chroma residual samples in the CU (1102), the scaling factors are used to scale the chroma residual samples in the CU (1103), and the reconstructed chroma samples are calculated by adding the chroma prediction samples in the CU and the scaled chroma residual samples (1104).

[0123] According to one or more embodiments of the second chroma sample reconstruction procedure, the predetermined intermediate luma prediction stage may include a decoder-side motion vector derivation (DMVR), a BD The bidirectional prediction module includes one or more of the following: Bi-Directional Optical Flow (OF), Combined Inter and Intra Prediction (CIIP), and DMVR, BDOF / PROF, and CIIP combined intra / inter prediction modules. In these embodiments, inter-predicted samples derived before the DMVR, BDOF / PROF, and CIIP combined intra / inter processes are used to derive scaling factors for the chroma residual.

[0124] 12 is a flowchart illustrating the workflow of the LMCS decoding process in one example of this embodiment of the second chroma sample reconstruction procedure, in which DMVR, BDOF, and CIIP are not applied to generate luma prediction samples for chroma scaling. Here, instead of waiting for the DMVR 1203, BDOF 1204, and / or luma intra prediction unit 1205 of CIIP to fully finish, the chroma residual scaling process 1208 can start as soon as prediction samples 1221 and 1222 based on the initial L0 and L1 luma predictions 1201 and 1202 are available.

[0125] In FIG. 12, in addition to the original averaging operation 1206, an additional averaging operation 1211 is required to combine the initial L0 predicted samples 1221 and L1 predicted samples 1222 before DMVR 1203, BDOF 1204, and / or CIIP 1205.

[0126] To reduce complexity, in a second example of this embodiment of the second chroma sample reconstruction procedure, the first L0 prediction sample may be used to derive the scaling factor for the chroma residual.

[0127] 13 is a flowchart showing the workflow of the LMCS decoding process in a second example of this embodiment of the second chroma sample reconstruction procedure, in which an initial uni-prediction signal is applied to generate luma prediction samples for chroma scaling. No additional averaging operations are required in addition to the original averaging operation 1306. The initial L0 prediction samples 1321 are used to derive scaling factors for the chroma residual before DMVR 1303, BDOF 1304, and / or CIIP 1305.

[0128] In a third example of this embodiment of the second chroma sample reconstruction procedure, one initial prediction signal (L0 or L1) is adaptively selected as the luma prediction sample used to derive the chroma residual scaling factor. In one possible implementation of this example, among the initial prediction signals (L0 or L1), the one whose reference picture has a smaller picture order count (POC) distance to the current picture is selected to derive the chroma residual scaling factor.

[0129] Another embodiment of the second chroma sample reconstruction procedure proposes to disable only DMVR and BDOF / PROF while enabling CIIP to generate inter-predicted samples used to determine the chroma residual scaling factor. Specifically, in this method, the inter-predicted samples derived before DMVR and BDOF / PROF are averaged and then combined with the intra-predicted samples of CIIP, and finally the combined predicted samples are used as predicted samples to determine the chroma residual scaling factor.

[0130] In yet another embodiment of the second chroma sample reconstruction procedure, it is proposed to disable only BDOF / PROF while maintaining DMVR and CIIP for generating the predicted samples used to determine the chroma residual scaling factor.

[0131] In yet another embodiment of the second chroma sample reconstruction procedure, it is proposed to maintain BDOF / PROF and CIIP while disabling DMVR when deriving the luma prediction samples used to determine the chroma residual scaling factor.

[0132] Furthermore, although the methods in the above embodiment of the second chroma sample reconstruction procedure are shown as being designed to reduce the latency of chroma prediction residual scaling, it should be noted that these methods may also be used to reduce the latency of luma prediction residual scaling. For example, all of these methods can also be applied to the PDRS method described in "Luminance Mapping Based on Prediction-Dependent Residual Scaling."

[0133] According to existing DMVR designs, to reduce computational complexity, the prediction samples used for DMVR motion refinement are generated using a 2-tap bilinear filter instead of the default 8-tap interpolation. After the refined motion is determined, the default 8-tap filter is applied to generate the final prediction samples for the current CU. Therefore, to reduce the chroma residual decoding latency caused by DMVR, it is proposed to use the luma prediction samples generated by the bilinear filter (the average of the L0 and L1 prediction samples if the current CU is bi-predicted) to determine the scaling factor for the chroma residual.

[0134] According to one chroma residual sample reconstruction procedure, one or more chroma residuals are extracted from the output of the bilinear filter of the Decoder-side Motion Vector Derivation (DMVR), as shown in Figure 14. A luma prediction sample value is selected (1401), one or more selected luma prediction sample values ​​are scaled to another plurality of luma prediction sample values ​​having the same bit depth as the original encoding bit depth of the input video (1402), the luma prediction sample value having the same bit depth as the original encoding bit depth of the input video is used to derive a scaling factor for decoding one or more chroma residual samples (1403), the scaling factor is used to scale the one or more chroma residual samples (1404), and the one or more chroma residual samples are reconstructed by adding the one or more scaled chroma residual samples and their corresponding chroma prediction samples (1405).

[0135] In one or more embodiments of the chroma residual sample reconstruction procedure, selecting one or more luma prediction sample values ​​from the output of the bilinear filter of the DMVR includes selecting L0 and L1 luma prediction samples from the output of the bilinear filter of the DMVR.

[0136] 15 is a flowchart illustrating the workflow of the LMCS decoding process in one such embodiment of the chroma residual sample reconstruction procedure. The L0 and L1 prediction samples 1521 and 1522 from the output of the bilinear filter 1512 component of the DMVR 1503 are fed to an average 1511 to derive a chroma residual scaling input 1523 to be used in chroma residual scaling 1507 to decode one or more chroma residual samples.

[0137] In these embodiments, there is a bit code depth issue. To save the internal storage size used by the DMVR, the intermediate L0 and L1 prediction samples generated by the bilinear filter of the DMVR are 10-bit precision. This differs from the representation bit depth of the immediate prediction samples of normal bi-prediction, which is equal to 14 bits. Therefore, due to their different precision, the intermediate prediction samples output from the bilinear filter cannot be directly applied to determine the chroma residual scaling factor.

[0138] To address this issue, we first propose to align the DMVR intermediate bit depth with the intermediate bi-depth used in regular motion compensated interpolation, i.e., increase the bit depth from 10 bits to 14 bits. Then, the existing averaging process applied to generate regular bi-predicted signals can be reused to generate corresponding predicted samples for determining the chroma residual scaling factor.

[0139] In one example of these embodiments, one or more selected luma prediction sample values ​​are combined with another plurality of luma prediction sample values ​​having the same bit depth as the original encoding bit depth of the input video. Adjusting to the input video bit depth includes increasing the internal bit depth of the L0 and L1 luma prediction samples from the output of the bilinear filter of the DMVR to 14 bits through left shifting, obtaining a 14-bit average luma prediction sample value by averaging the 14-bit shifted L0 and L1 luma prediction sample values, and converting the 14-bit average luma prediction sample value by changing the internal bit depth of the 14-bit average luma prediction sample value to the original coding bit depth of the input video through right shifting.

[0140] More specifically, in this example, the saturation scaling factor is determined by the steps described in the box immediately following this paragraph.

number

[0141] In another embodiment of the chroma residual sampling reconstruction procedure, selecting one or more luma prediction sampling values ​​from the output of the bilinear filter of the DMVR and adjusting the one or more selected luma prediction sampling values ​​to another plurality of luma prediction sampling values ​​at the same bit depth as the original encoding bit depth of the input video includes selecting one luma prediction sampling from the L0 and L1 luma prediction samplings from the output of the bilinear filter of the DMVR, adjusting the one selected luma prediction sampling by changing the internal bit depth of the one selected luma prediction value to the original encoding bit depth of the input video through shifting, and using the adjusted luma prediction sampling at the same bit depth as the original encoding bit depth of the input video as the luma prediction sampling.

[0142] FIG. 16 is a flowchart showing the workflow of the LMCS decoding process in one such alternative embodiment of the chroma residual sample reconstruction procedure. The L0 prediction samples 1621 from the output of the linear filter 1612 component are used in chroma residual scaling 1607 to decode one or more chroma residual samples.

[0143] In one example of one such alternative embodiment of the chroma residual sample reconstruction procedure, the chroma scaling factor is determined by shifting the luma samples output from the bilinear filter to the original coding bit depth of the input video, as described in the box immediately following this paragraph.

number

[0144] According to the second chroma residual sample reconstruction procedure, as shown in FIG. 17, one or more luma reference sample values ​​are selected from a reference picture (1701), the one or more selected luma reference sample values ​​are converted to luma sample values ​​(1702), the converted luma sample values ​​are used to derive scaling factors (1703), the scaling factors are used to scale one or more chroma residual samples (1704), and one or more chroma residual samples are reconstructed by adding one or more scaled chroma residual samples and their corresponding chroma prediction samples (1705).

[0145] In one or more embodiments of the second chroma residual sample reconstruction procedure, selecting one or more luma reference sample values ​​from the reference picture and converting the one or more selected luma reference sample values ​​to luma sample values ​​includes obtaining both L0 and L1 luma reference sample values ​​from the L0 and L1 reference pictures and averaging the L0 and L1 luma reference sample values ​​as the converted luma sample values.

[0146] In another embodiment of the second chroma residual sample reconstruction procedure, selecting one or more luma reference samples from the reference picture and converting the one or more selected luma reference samples to luma sample values ​​includes selecting one luma reference sample value from the L0 and L1 luma reference sample values ​​from the L0 and L1 reference pictures, and using the one selected luma reference sample value as the converted luma sample value.

[0147] According to the existing LMCS design, the reconstructed luminance samples adjacent to the 64×64 region where the current CU is located are used to calculate the chroma residual scaling factor of the CUs within the region. Further, one clipping operation, i.e., Clip1(), is applied to clip the luminance adjacent samples reconstructed to the dynamic range of the internal bit depth (i.e., the range of [0, (1<<bitDepth)-1]) before the average is calculated.

[0148] Specifically, this method first fetches 64 left adjacent luminance samples and 64 upper adjacent luminance samples of the corresponding 64×64 region to which the current CU belongs, then calculates the average of the left adjacent samples and the upper adjacent samples, i.e., avgY, and finds the segment index YIdx of avgY in the LMCS piecewise linear model, and finally derives the chroma residual CScale Inv = cScaleInv[YIdx]. Inv = cScaleInv[YIdx].

[0149] Specifically, in the current VVC draft, the method for deriving the corresponding average luminance is as follows. The application of the Clip1() operation is shown in prominent font face and size.: <�

Number

[0150] However, in the reconstruction process, after adding the predicted samples to the residual samples of one CU, the resulting sample values are already clipped to the dynamic range of the internal bit depth. This means that all adjacent reconstructed luminance samples around the current 64×64 region are guaranteed to be within the range of the internal bit depth. Therefore, their average, i.e., avgY, cannot also exceed this range. As a result, the existing clipping (i.e., Clip1()) is for the corresponding chroma residual scaling To further reduce the complexity and memory requirements of the LMCS design, it is proposed to eliminate the clipping operation when calculating the average of neighboring reconstructed luma samples to derive the chroma residual scaling factor.

[0151] According to a first aspect of the present application, as shown in FIG. 18, a plurality of reconstructed luma samples from a first predetermined area adjacent to a second predetermined area in which the CU is located are selected during decoding of the CU (1801), an average of the plurality of reconstructed luma samples is calculated (1802), and the average of the plurality of reconstructed luma samples is directly used without clipping when deriving a chroma residual scaling factor for decoding the CU (1803).

[0152] In one or more embodiments of the first aspect of the present application, the average of the plurality of reconstructed luminance samples is an arithmetic mean of the plurality of reconstructed luminance samples.

[0153] In one or more embodiments of the first aspect of the present application, directly using an average of multiple reconstructed luma samples without clipping when deriving a chroma residual scaling factor for decoding a CU includes identifying a segment index for the average in a predetermined piecewise linear model, and deriving a luma residual scaling factor for decoding the CU based on a gradient of the linear model of the segment.

[0154] In one or more embodiments of the first aspect of the present application, a plurality of reconstructed luma samples within a first predetermined region are generated by generating luma prediction samples and luma residual samples within the first predetermined region, adding the luma residual samples to the luma prediction samples, and clipping the added luma samples to a dynamic range of an encoding bit depth.

[0155] In one or more embodiments of the first aspect of the present application, the plurality of reconstructed luma samples are a plurality of forward-mapped inter-luma reconstructed samples.

[0156] In one or more embodiments of the first aspect of the present application, the second predetermined region is a 64x64 region in which the CU is located.

[0157] 19, the first predetermined region may include the top-neighboring samples in a 1×64 region 1902 immediately above the second predetermined region 1904. Alternatively or additionally, the first predetermined region may include the left-neighboring samples in a 64×1 region 1903 immediately to the left of the second predetermined region 1904.

[0158] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media, including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communications protocol. As such, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media or (2) communication media such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for execution of the implementations described herein. A computer program product may include computer-readable media.

[0159] Furthermore, the above method can be implemented using ASICs (application specific integrated circuits), D The methods may be implemented using an apparatus including one or more circuits, including digital signal processors (SPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components. The apparatus may use the circuits in combination with other hardware or software components to perform the methods. Each module, sub-module, unit, or sub-unit disclosed above may be implemented at least in part using one or more circuits.

[0160] The present disclosure may include dedicated hardware implementations such as application-specific integrated circuits, programmable logic arrays, and other hardware devices. The hardware implementations may be configured to implement one or more of the methods described herein. Examples that may include various implementations of devices and systems may broadly include various electronic and computing systems. One or more examples described herein may implement functions using two or more specific interconnected hardware modules or devices with associated control and data signals that can communicate between, through, or as part of an application-specific integrated circuit. Thus, the disclosed devices or systems may encompass software, firmware, and hardware implementations. Terms such as "module," "sub-module," "circuit," "sub-circuit," "unit," or "sub-unit" may include memory (shared, dedicated, or group) that stores code or instructions that can be executed by one or more processors. A module, as used herein, may include one or more circuits, with or without stored code or instructions. A module or circuit may include one or more connected components.

[0161] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure in accordance with its general principles, such departures from the disclosure coming within known or customary practice in the art. The specification and embodiments are intended to be exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0162] It will be understood that the present disclosure is not limited to the exact examples described above and illustrated in the accompanying drawings, and that various modifications and changes can be made thereto without departing from the scope thereof, which is intended to be limited only by the appended claims.

[0163] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 001,229, filed March 27, 2020. The entire disclosure of the foregoing application is incorporated herein by reference in its entirety for all purposes.

Claims

1. obtaining an average of a plurality of reconstructed luminance samples adjacent to a predetermined region in which the coding unit is located; deriving a chroma residual scaling factor directly using an average of a plurality of said reconstructed luma samples without clipping; Forming a bitstream with LMCS (luma mapping with chroma scaling) related information; Video coding methods.

2. The video encoding method of claim 1 , wherein the average of the reconstructed luma samples is an arithmetic mean of the reconstructed luma samples.

3. Deriving the chroma residual scaling factor directly using an average of a plurality of the reconstructed luma samples without clipping includes: Identifying an average segment index in a predetermined piecewise linear model; deriving the chroma residual scaling factor based on the segment index; 2. The video encoding method of claim 1, comprising:

4. The plurality of reconstructed luminance samples comprises: generating luma prediction samples and luma residual samples; adding the luma residual sample to the luma prediction sample to obtain a luma sum sample; clipping the summed luminance samples to a dynamic range of an encoding bit depth, the encoding bit depth being 14 bits; It is generated by 2. The video encoding method of claim 1.

5. The video encoding method of claim 1 , wherein the reconstructed luma samples are derived based on forward-mapped inter-predicted samples.

6. The video encoding method of claim 1 , wherein the predetermined area is a 64×64 area in which the encoding unit is located.

7. 7. The video encoding method of claim 6, wherein the regions adjacent to the predetermined region include a 1x64 region immediately above the predetermined region and a 64x1 region immediately to the left of the predetermined region.

8. one or more processors; a non-transitory storage device coupled to one or more of said processors; A plurality of programs stored in the non-transitory storage device, When executed by one or more of said processors, causes a computing device to perform the video encoding method of any one of claims 1 to 7, generating a bitstream and storing said bitstream in said non-transitory storage device. Several programs and 1. A computing device comprising:

9. A non-transitory computer-readable storage medium storing a plurality of programs, The programs, when executed by one or more processors, cause a computing device to perform the video encoding method of any one of claims 1 to 7. , generating a bitstream and transmitting the bitstream; A non-transitory computer-readable storage medium.

10. 1. A method for storing a bitstream in a non-transitory storage device, comprising: the bitstream is generated by an encoding method; The method of storing includes: storing the bitstream; The encoding method comprises: obtaining an average of a plurality of reconstructed luminance samples adjacent to a predetermined region in which the coding unit is located; deriving a chroma residual scaling factor directly using an average of a plurality of said reconstructed luma samples without clipping; forming the bitstream with LMCS (luma mapping with chroma scaling) related information; Including, How the bitstream is stored.

Images