Video encoding method, computing device, program, and method for storing bitstreams - Patents.com

Abstract

To provide a system and a method for executing coding of a video by using a prediction-dependent residual scaling for a coded unit.SOLUTION: A block-based hybrid video encoder 100 selects a plurality of reconfiguration luminance samples from a first specific region next to a second specific region in which a coded unit is located; calculates the average of the plurality of reconfiguration luminance samples; derives a chroma residual scaling factor by directly using the average of the plurality of reconfiguration luminance samples without clipping; and coding an LMCS (luma mapping with chroma scaling)-related information to a bit stream.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to video encoding and compression, and more particularly to systems and methods for performing video encoding using prediction-dependent residual scaling for a coding unit. [Background technology]

[0002] This section provides background information related to the present disclosure. The information contained in this section should not necessarily be construed as prior art.

[0003] To compress the video data, any of a variety of video encoding techniques may be used. The video encoding may be performed according to one or more video encoding standards. Some exemplary video encoding standards include versatile video coding (VVC), joint exploration test model (JEM) coding, high-efficiency video coding (H.265 / HEVC), advanced video coding (H.264 / AVC), and moving picture experts group (MPEG) coding.

[0004] Video coding generally employs prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit the redundancy inherent in a video image or sequence. One goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation of video quality.

[0005] The first version of the HEVC standard was completed in October 2013, and provides approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard H.264 / MPEG AVC. While the HEVC standard provides significant coding improvements over its predecessors, there is evidence that better coding efficiency can be achieved with additional coding tools than HEVC. Based on this, VCEG and MPEG have begun work on exploring new coding techniques for future video coding standardization. The Joint Video Exploration Team (JVET) was formed by ITU-T VECG and ISO / IEC MPEG in October 2015 to begin significant research into advanced technologies that could enable substantial improvements in coding efficiency. One reference software, called the joint exploration model (JEM), is It was maintained by JVET by integrating some additional coding tools on top of the HEVC Test Model (HM).

[0006] In October 2017, a joint call for proposals (CfP) for video compression with capabilities beyond HEVC was issued by ITU-T and ISO / IEC. In April 2018, 23 CfP responses were received and evaluated at the 10th JVET meeting, and the results demonstrated an improvement in compression efficiency beyond HEVC of about 40%. Based on these evaluation results, a new project was launched to develop a new generation video coding standard named Versatile Video Coding (VVC). In the same month, one reference software called VVC Test Model (VTM) was established to demonstrate a reference implementation of the VVC standard.

[0007] Prediction methods utilized in video coding typically involve performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video data, and are typically associated with block-based video coding. Like HEVC, VVC is built on a block-based hybrid video coding framework.

[0008] In block-based video coding, the input video signal is processed block by block. For each block (also known as a coding unit (CU)), spatial prediction and / or temporal prediction may be performed. In newer video coding standards, such as the current VVC design, blocks may be further divided based on multi-type tree structures, including not only quad-trees, but also binary and / or ternary trees. This allows for better adjustment of various local characteristics.

[0009] Spatial prediction (also known as "intra prediction") uses pixels from samples of already coded neighboring blocks (called reference samples) in the same video picture / slice to predict the current block. Spatial prediction reduces the spatial redundancy inherent in video signals.

[0010] During the decoding process, the video bitstream is first entropy decoded in an entropy decoding unit. The coding mode and prediction information are sent to either a spatial prediction unit (when intra-coded) or a temporal prediction unit (when inter-coded) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit and an inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then added. The reconstructed block may further undergo in-loop filtering before being stored in a reference picture store. The reconstructed video in the reference picture store is then sent to drive a display device and used to predict future video blocks.

[0011] In newer video coding standards, such as the current VVC design, a coding tool called luma mapping with chroma scaling (LMCS) can be applied before in-loop filtering. LMCS aims to adjust the dynamic range of the input signal to improve coding efficiency. Summary of the Invention [Problem to be solved by the invention]

[0012] However, the current design of LMCS uses different domain mappings in various decoding modules, which incurs computational complexity and additional on-chip memory requirements. In addition, the current design of LMCS uses different luma prediction sample values ​​to derive luma and chroma scaling factors, which introduces additional complexity. In addition, the current design of LMCS requires postponing the reconstruction of chroma residual samples until after successful completion of luma prediction samples, which increases the latency of the reconstruction of chroma residual samples, which is inconvenient for complex inter-prediction schemes such as Decoder-side Motion Vector Derivation (DMVR), Bi-Directional Optical Flow (BDOF), and Combined Inter and Intra Prediction (CIIP). This requires successful completion of sequential application of the mode encoding tools. The current design of the LMCS also performs unnecessary clipping operations during the chroma residual scaling factor derivation process, further increasing computational complexity and requiring additional on-chip memory. [Means for solving the problem]

[0013] This section provides a general overview of the disclosure, but is not an exhaustive disclosure of its entire scope or any of its features.

[0014] According to a first aspect of the present application, a plurality of reconstructed luma samples from a first predetermined region adjacent to a second predetermined region in which the CU is located are selected during decoding of the CU, an average of the plurality of reconstructed luma samples is calculated, and the average of the plurality of reconstructed luma samples is directly used without clipping when deriving a chroma residual scaling factor for decoding the CU.

[0015] According to a second aspect of the present application, a computing device includes one or more processors. The present invention relates to a computing device including a processor, a memory, and a plurality of programs stored in the memory, the programs being configured to, when executed by the one or more processors, cause the computing device to perform the operations described above in the first aspect of the present application.

[0016] According to a third aspect of the present application, a non-transitory computer-readable storage medium stores a number of programs for execution by a computing device having one or more processors, the programs, when executed by the one or more processors, causing the computing device to perform the operations described above in the first aspect of the present application. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram illustrating an example block-based hybrid video encoder that may be used in many video coding standards. [Diagram 2] FIG. 1 is a block diagram illustrating an example video decoder that may be used in many video encoding standards. [Diagram 3] FIG. 2 is a diagram of block partitioning in a multi-type tree structure that may be used in many video coding standards. [Figure 4] 13 is a flowchart showing a decoding process when LMCS is applied. [Diagram 5] FIG. 1 is an explanatory diagram of the BDOF process. [Figure 6] 13 is a flowchart showing the workflow of chroma residual scaling in LMCS when DMVR, BDOF, and CIIP are all enabled. [Figure 7] FIG. 1 is a flowchart showing the steps of the prediction dependent residual scaling (PDRS) procedure. [Figure 8] 1 is a flowchart showing a workflow of a decryption process when a PDRS procedure is applied in LMCS processing. [Figure 9]FIG. 1 is an illustration of the residual mapping error caused by simply using predicted samples to derive the scaling factor. [Figure 10] 13 is a flow chart showing steps of a chroma sample reconstruction procedure. [Figure 11] 11 is a flow chart showing steps of a second chroma sample reconstruction procedure. [Figure 12] 13 is a flowchart illustrating a workflow of the LMCS decoding process in an example of a second chroma sample reconstruction procedure in which DMVR, BDOF, and CIIP are not applied to generate luma prediction samples for chroma scaling. [Figure 13] A flowchart showing the workflow of the LMCS decoding process in a second example of the second chroma sample reconstruction procedure, in which an initial uni-prediction signal is applied to generate luma prediction samples for chroma scaling. [Figure 14] 13 is a flow chart showing steps of a chroma residual sample reconstruction procedure. [Figure 15] 13 is a flowchart illustrating a workflow of the LMCS decoding process in one or more embodiments of the chroma residual sample reconstruction procedure. [Figure 16] 13 is a flowchart illustrating the workflow of the LMCS decoding process in one alternative embodiment of the chroma residual sample reconstruction procedure. [Figure 17] 11 is a flow chart showing steps of a second chroma residual sample reconstruction procedure. [Figure 18] 1 is a flow chart showing steps of a first aspect of the present disclosure. [Figure 19] FIG. 1 is a diagram of the areas involved in an embodiment of the first aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] A set of exemplary and non-limiting embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. Variations in structure, method, or function may be implemented by those skilled in the art based on the examples presented herein, and all such variations are included within the scope of the present disclosure. If no contradiction exists, the teachings of different embodiments may be combined with each other, but need not be.

[0019] The terms used in this disclosure are not intended to limit the disclosure, but to describe specific examples. The singular form "a" and the like used in this disclosure and the appended claims also refer to the plural form unless the context clearly indicates otherwise. It should be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0020] Terms such as "first," "second," and "third" may be used herein to describe various pieces of information, but it should be understood that the information should not be limited by these terms. These terms are used only to distinguish one category of information from another category of information. For example, the first information can be referred to as the second information, and similarly, the second information can be referred to as the first information, without departing from the scope of this disclosure. As used herein, the terms "if," "when," "in response to," and the like, may be understood to mean "when," "in," or "in response to," depending on the context.

[0021] Throughout this specification, the singular or plural "embodiments," "another embodiment," and the like means that one or more particular features, configurations, etc. described in connection with an embodiment are included in at least one embodiment of the present disclosure. Thus, appearances of phrases such as "in one embodiment," "in an embodiment," "in another embodiment," and the like throughout this specification do not necessarily all refer to the same embodiment. Furthermore, particular features, structures, or characteristics in one or more embodiments may be combined in any suitable manner.

[0022] Conceptually, many video coding standards, including those discussed above in the Background section, are similar, for example, virtually all video coding standards use block-based processing and share similar video coding block diagrams to achieve video compression.

[0023] FIG. 1 shows a block diagram of an exemplary block-based hybrid video encoder 100 that may be used in many video coding standards. In the encoder 100, a video frame is divided into multiple video blocks for processing. For a given video block, a prediction is formed based on either an inter-prediction technique or an intra-prediction technique. In inter-prediction, one or more predictors are formed based on pixels from a previously reconstructed frame through motion estimation and motion compensation. In intra-prediction, a predictor is formed based on reconstructed pixels of a current intra-frame. Through a mode decision, the best predictor may be selected to predict the current block.

[0024] A prediction residual, which represents the difference between the current video block and its predictor, is sent to a transform circuit 102. The transform coefficients are then sent from the transform circuit 102 to a quantization circuit 104 for entropy reduction. The quantized coefficients are then provided to an entropy coding circuit 106 to generate a compressed video bitstream. As shown in FIG. 1, prediction related information 110 from an inter prediction circuit and / or an intra prediction circuit 112, such as video block partition information, motion vectors, reference picture indexes, and intra prediction modes, are also provided via the entropy coding circuit 106 and stored in the compressed video bitstream 114.

[0025] In the encoder 100, decoder-related circuitry is also required to reconstruct pixels for prediction. First, a prediction residual is reconstructed via an inverse quantization circuit 116 and an inverse transform circuit 118. This reconstructed prediction residual is combined with a block predictor 120 to generate unfiltered reconstructed pixels for the current video block.

[0026] Temporal prediction (also called "inter prediction" or "motion compensated prediction") uses reconstructed pixels from an already coded video picture to predict a current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. Also, if multiple reference pictures are supported, one reference picture index is additionally sent, which is used to identify which reference picture in the reference picture store the temporal prediction signal comes from.

[0027] After spatial and / or temporal prediction is performed, an intra / inter mode decision circuit 121 in the encoder 100 selects the best prediction mode, for example based on a rate-distortion optimization method. The block predictor 120 is then subtracted from the current video block, and the resulting prediction residual is decorrelated using the transform circuit 102 and the quantization circuit 104. The resulting quantized residual coefficients are inverse quantized by the inverse quantization circuit 116 and inverse transformed by the inverse transform circuit 118 to form a reconstructed residual, which is then added back to the prediction block to form the reconstructed signal for the CU. Additional filter techniques, such as a deblocking filter, sample adaptive offset (SAO), and / or adaptive in-loop filter (ALF), may be used to filter the current video block. An additional in-loop filter 115 may be applied to the reconstructed CU before it is placed in a reference picture store in a picture buffer 117 and used to encode future video blocks. To form the output video bitstream 114, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit 106 where they are further compressed and packed to form a bitstream.

[0028] For example, deblocking filters are available in AVC, HEVC, as well as the current version of VVC. In HEVC, an additional in-loop filter called sample adaptive offset (SAO) is defined to further improve coding efficiency. In the current VVC standard, yet another in-loop filter called adaptive loop filter (ALF) is under active consideration and is likely to be included in the final standard.

[0029] These in-loop filter operations are optional. Performing these operations helps improve coding efficiency and visual quality. They can also be turned off as a decision rendered by the encoder 100 to save computational complexity.

[0030] Note that intra prediction is typically based on unfiltered reconstructed pixels, while inter prediction is based on filtered reconstructed pixels if these filter options are turned on by the encoder 100.

[0031] FIG. 2 is a block diagram illustrating an exemplary video decoder 200 that may be used in many video coding standards. The decoder 200 is similar to the reconstruction-related section present in the encoder 100 of FIG. 1. In the decoder 200 (FIG. 2), an input video bitstream 201 is first decoded through entropy decoding 202 to derive quantized coefficient levels and prediction-related information. The quantized coefficient levels are then processed through inverse quantization 204 and inverse transform 206 to obtain a reconstructed prediction residual. A block prediction mechanism implemented in intra / inter mode selection 212 is configured to perform either intra prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixels is added to the reconstructed prediction residual from the inverse transform 206 and the prediction output generated by the block prediction mechanism using an adder 214. and summing it up.

[0032] The reconstructed blocks may further pass through an in-loop filter 209 before being stored in a picture buffer 213, which acts as a reference picture store. The reconstructed video in the picture buffer 213 may then be sent out to drive a display device, as well as used to predict future video blocks. In situations where the in-loop filter 209 is turned on, a filtering operation is performed on these reconstructed pixels to derive the final reconstructed video output 222.

[0033] In video coding standards such as HEVC, blocks may be divided based on a quadtree. New video coding standards such as current VVC adopt more division methods, and coding tree units (CTUs) may be divided into CUs based on quadtrees, binary trees or ternary trees to adapt to changing local characteristics. The separation of CUs, prediction units (PUs) and transform units (TUs) does not exist in most coding modes in current VVC, and each CU is always used as a basic unit for both prediction and transformation without further division. However, in some specific coding modes such as sub-division coding modes, each CU may still contain multiple TUs, and in the multi-type tree structure, one CTU is first divided by a quadtree structure. Then, each quadtree leaf node may be further divided by binary and ternary tree structures.

[0034] 3 shows five partition types used in current VVC: 4-partition 301, horizontal 2-partition 302, vertical 2-partition 303, horizontal 3-partition 304, and vertical 3-partition 305. In a situation where a multi-type tree structure is utilized, one CTU is first partitioned by a quadtree structure. Then, each quadtree leaf node can be further partitioned by a 2-part tree structure and a 3-part tree structure.

[0035] Using one or more of the example block partitions 301, 302, 303, 304, or 305 of Figure 3, spatial prediction and / or temporal prediction may be performed using the configuration shown in Figure 1. Spatial prediction (or "intra prediction") uses pixels from samples of already coded neighboring blocks (called reference samples) in the same video picture / slice to predict a current video block. Spatial prediction reduces spatial redundancy inherent in video signals.

[0036] Newer video coding standards, such as VVC, now add a new coding tool called Luma Mapping with Chroma Scaling (LMCS), which is applied before the loop filters (e.g., deblocking filter, SAO, and ALF).

[0037] In general, LMCS has two main modules: the first is an in-loop mapping of the luma component based on an adaptive piecewise linear model, and the second is a luma-dependent chroma residual scaling.

[0038] Figure 4 shows a modified decoding process where LMCS is applied. In Figure 4, several blocks represent the decoding module taking place in the mapped domain, including entropy decoding 401, inverse quantization 402, inverse transform 403, luma intra prediction 404, and luma sample reconstruction 405 (i.e., adding luma prediction samples Y'pred and luma residual samples Y'res to generate reconstructed luma samples Y'recon). Several other blocks include motion compensated prediction 409, chroma intra prediction 412, chroma sample reconstruction 413 (i.e., adding luma prediction samples Cpred and luma residual samples Cres to generate reconstructed chroma samples Crecon), and in-loop filtering 414. 4 illustrates the decoding modules performed in the original (i.e., unmapped) domain, including LMCS 407 (which encompasses deblocking, SAO, and ALF). A further group of blocks represents new operation modules introduced by LMCS, including forward mapping 410 and backward (or inverse) mapping 406 of luma samples, and chroma residual scaling 411. Also, as shown in FIG. 4, all reference pictures stored in the decoded picture buffers (DPBs) 408 (for luma) and 415 (for chroma) are in the original domain.

[0039] The in-loop mapping of LMCS aims to adjust the dynamic range of the input signal to improve coding efficiency. The in-loop mapping of luma samples in the existing LMCS design is constructed based on two mapping functions: one forward mapping function FwdMap and one corresponding inverse mapping function InvMap. The forward mapping function is signaled from the encoder to the decoder using one piecewise linear model with 16 equally sized pieces. The inverse mapping function can be derived directly from the forward mapping function and therefore does not need to be signaled.

[0040] The parameters of the luma mapping model are signaled at the slice level. A presence flag is first signaled to indicate whether a luma mapping model should be signaled for the current slice. If a luma mapping model is present in the current slice, the corresponding piecewise linear model parameters are further signaled. Based on the piecewise linear model, the dynamic range of the input signal is divided into 16 segments having equal sizes in the original domain, and each segment is mapped to a corresponding segment. For a given segment in the original domain, its corresponding segment in the mapped domain may have the same or different sizes. The size of each segment in the mapped domain is indicated by the number of codewords (i.e., the mapped sample values) of that segment. For each segment in the original domain, the linear mapping parameters can be derived based on the number of codewords in the corresponding segment in the mapped domain. For example, when the input is in 10-bit depth, each of the 16 segments in the original domain has 64 pixel values, and if each of the segments in the mapped domain also has 64 codewords assigned to it, it indicates a simple one-to-one mapping (i.e., a mapping in which each sample value is unchanged). The signaled number of codewords for each segment in the mapped domain is used to calculate a scaling factor and adjust the mapping function accordingly for that segment. Additionally, at the slice level, another LMCS control flag is signaled to enable / disable LMCS for the slice.

[0041] For each segment, the corresponding piecewise linear model is defined as described in the box immediately following this paragraph:

number

[0042] As shown in FIG. 4, it is necessary to operate in two different domains during the LMCS process. For each CU coded via an inter prediction mode ("inter CU"), its motion compensation prediction is performed in the original domain. However, since the reconstruction of the luma component (i.e., the addition of luma prediction samples and luma residual samples) is performed in the mapped domain, the motion compensated luma prediction Ypred needs to be mapped from the original domain to a value Y'pred in the mapped domain via a forward mapping function 410, i.e., Y'pred=FwdMap(Ypred), and then Y'pred is used for pixel reconstruction 405. On the other hand, for each CU coded via an intra prediction mode ("intra CU"), after intra prediction 404 is performed in the mapped domain (as shown in FIG. 4), Y'pred is used for pixel reconstruction 405, so no mapping of prediction samples is required. Finally, after generating the reconstructed luma samples Y'recon, a backward mapping function 406 is applied to convert the reconstructed luma samples Y'recon to Yrecon in the original domain, and then proceeds to the luma DPB 408, i.e., Yrecon=InvMap(Y'recon). Unlike the forward mapping of predicted samples 410, which only needs to be applied to inter CUs, the backward mapping of reconstructed samples 406 needs to be applied to both inter and intra CUs.

[0043] In summary, on the decoder side, an in-loop luma mapping of the current LMCS is performed such that the luma prediction samples Ypred are first transformed, if necessary, into the mapped domain: Y'pred=FwdMap(Ypred). The mapped prediction samples are then summed with the decoded luma residuals to form reconstructed luma samples in the mapped domain: Y'recon=Y'pred+Y'res. Finally, an inverse mapping is applied to transform the reconstructed luma samples Y'recon back to the original domain: Yrecon=InvMap(Y'recon). On the encoder side, the luma residuals are coded in the mapped domain, so they are generated as the difference between the mapped original luma samples and the mapped luma prediction samples: Y'res=FwdMap(Y'recon). p(Yorg) - FwdMap(Ypred).

[0044] The second step of LMCS, luma-dependent chroma residual scaling, is designed to compensate for the interaction of quantization precision between a luma signal and its corresponding chroma signal when in-loop mapping is applied to the luma signal. Whether chroma residual scaling is enabled or disabled is also signaled in the slice header. When luma mapping is enabled and binary tree partitioning of luma and chroma components is disabled for the current slice, an additional flag is signaled to indicate whether luma-dependent chroma residual scaling is applied or not. When luma mapping is not used or binary tree partitioning is enabled for the current slice, luma-dependent chroma residual scaling is always disabled. Additionally, chroma residual scaling is always disabled for CUs that contain four or fewer chroma samples.

[0045] For both intra and inter CUs, the scaling parameters used to scale the chroma residuals depend on the average of the corresponding mapped luma prediction samples. The scaling parameters are derived as described in the box immediately following this paragraph:

number

[0046] 4 also illustrates the computation of the average of luma prediction samples for luma-dependent chroma residual scaling. For inter CUs, the forward-mapped luma prediction Y'pred is fed to chroma residual scaling 411 together with the scaled chroma residual CresScale to derive a chroma residual Cres which is fed to chroma reconstruction 413 together with the chroma prediction Cpred to derive a reconstructed chroma value Crecon. For intra CUs, intra prediction 404 generates Y'pred already in the mapped domain and is fed to chroma residual scaling 411 as in the inter CU case.

[0047] Unlike luma mapping, which is performed on a sample-by-sample basis, CScaleInv scales chroma Fixed for the entire CU. If CScaleInv is given, chroma residual scaling is applied as described in the box immediately following this paragraph.

number

[0048] In the current video coding standard such as VVC, new coding tools are introduced. Some examples of new coding tools are BDOF (Bi-Directional Optical Flow), DMV These are Decoder-side Motion Vector Refinement (R), Combined Inter and Intra Prediction (CIIP), Affine mode, and Prediction Refinement with Optical Flow (PROF) for Affine mode.

[0049] In the current VVC, BDOF (bi-directional optical flow) is used as a bidirectional predictive coding block. is applied to refine the lock prediction samples.

[0050] Figure 5 is an illustration of the BDOF process. BDOF is a sample-by-sample motion refinement performed in block-based motion compensation prediction when bidirectional prediction is used. The motion refinement (vx,vy) of each 4x4 subblock 501 is calculated by minimizing the difference between the predicted samples 502 and 503 of reference picture list 0 (L0) and reference picture list 1 (L1) after BDOF has been applied within one 6x6 window Ω around the subblock.

[0051] Specifically, the motion refinement values ​​(vx,vy) are derived as described in the box immediately following this paragraph.

number

[0052] The values ​​S1, S2, S3, S5, and S6 in the previous box are written in the box immediately following this paragraph. Further calculations are performed as described above.

number

[0053]

number

number

[0054] Based on the motion refinement derived as described in the box immediately following paragraph

[0072] above, the final bi-prediction samples for the CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the optical flow model, as shown in the box immediately following this paragraph.

[0055] Based on the above-mentioned bit depth control method, it is ensured that the maximum bit depth of intermediate parameters in the entire BDOF process does not exceed 32 bits, and the maximum input to the multiplier is within 15 bits, i.e., one 15-bit multiplier is sufficient for BDOF implementation.

number

[0056] DMVR is a bi-prediction technique used for merging blocks with two initially signaled MVs, which can be further refined by using bilateral matching prediction.

[0057] Specifically, DMVR uses bilateral matching to derive the motion information of the current CU by finding the best match between two blocks along the motion trajectory of the current CU in two different reference pictures. The cost function used in the matching process is the low subsampled sum of absolute difference (SAD). After the matching process is performed, the refined MV is used for motion compensation in the prediction stage, i.e., temporal motion vector prediction for subsequent pictures, and the unrefined MV is used for motion vector prediction between the motion vector of the current CU and its spatially adjacent motion vectors.

[0058] Under the assumption of continuous motion trajectories, the motion vectors MV0 and MV1 pointing to two reference blocks are proportional to the temporal distances between the current picture and the two reference pictures, i.e., TD0 and TD1. As a special case, when the current picture is temporally between the two reference pictures and the temporal distances from the current picture to the two reference pictures are the same, the bilateral matching becomes mirror-based bidirectional MV.

[0059] In current VVC, inter prediction and intra prediction methods are used in a hybrid video coding scheme, and each PU is only allowed to choose inter prediction or intra prediction to exploit correlation in either the time domain or the spatial domain, but never in both. However, as pointed out in previous literature, the residual signals generated by inter predicted blocks and intra predicted blocks may present very different characteristics from each other. Therefore, if the two kinds of prediction can be efficiently combined, one more accurate prediction can be expected to reduce the energy of the prediction residual and improve the coding efficiency. In addition, in video content, the movement of moving objects may be complex. For example, there may be areas that contain both old content (e.g., objects included in the previously coded picture) and new content that appears (e.g., objects different from the previously coded picture). In such a scenario, neither inter prediction nor intra prediction can provide an accurate prediction of the current block.

[0060] To further improve prediction efficiency, the VVC standard considers 1 A combined inter and intra prediction (CIIP) is adopted, which combines inter prediction and intra prediction of two CUs. Specifically, for each merged CU, one additional flag is signaled to indicate whether CIIP is enabled for the current CU. When the flag is equal to 1, the CIIP applies only planar mode to generate intra prediction samples of luma and chroma components. In addition, equal weights (i.e., 0.5) are applied to average the inter prediction samples and intra prediction samples as the final prediction samples of the CIIP CU.

[0061] VVC also supports affine modes for motion compensated prediction. In HEVC, for motion compensated prediction, only translational motion model is applied. In the real world, there are many kinds of motion, e.g., zoom in / zoom out, rotation, perspective motion, and other irregular motion. In VVC, affine motion compensated prediction is applied by signaling one flag for each inter-coded block to indicate whether affine motion model is applied for translational motion or inter prediction. In the current VVC design, two affine modes are supported for one affine coded block, including 4-parameter affine mode and 6-parameter affine mode.

[0062] The four-parameter affine model has the following parameters: two parameters for horizontal and vertical translational motion, respectively, one parameter for zoom motion, and one parameter for rotational motion in both directions. The horizontal zoom parameter is the same as the vertical zoom parameter. The horizontal rotation parameter is equal to the vertical rotation parameter. To achieve better adaptation of the motion vectors and affine parameters, in VVC these affine parameters are transformed into two MVs (also called control point motion vectors (CPMVs)) located at the top left and top right corners of the current block. The affine motion field of a block is described by two control point MVs (V0, V1).

[0063] Based on the movement of the control points, the motion field (vx, vy) of one affine coded block is calculated as described in the box immediately following this paragraph.

number

[0064] The 6-parameter affine mode has two parameters for horizontal and vertical translational motion, one parameter for zoom motion, and one parameter for horizontal rotational motion, one parameter for zoom motion, and one parameter for vertical rotational motion. The 6-parameter affine motion model is coded with 3 MVs in 3 CPMVs.

[0065] The three control points of a six-parameter affine block are located at the top-left, top-right, and bottom-left corners of the block. Movements at the top-left control point are associated with translational movements, movements at the top-right control point are associated with horizontal rotation and zoom movements, and movements at the bottom-left control point are associated with vertical rotation and zoom movements. Compared to the four-parameter affine motion model, the six-parameter horizontal rotation and zoom movements may not be the same as their vertical counterparts.

[0066] Assuming that (V0, V1, V2) are the MVs of the top-left, top-right, and bottom-left corners of the current block, the motion vectors (vx, vy) of each sub-block are derived using the three MVs at the control points, as described in the box immediately following this paragraph.

number

[0067] In order to improve the accuracy of affine motion compensation, prediction refinement using PROF (Prediction Refinement with Optical Flow) is currently being considered in the current VVC, which refines subblock-based affine motion compensation based on an optical flow model. Specifically, after performing sub-block-based affine motion compensation, the luma prediction samples of one affine block are refined by one sample refinement value derived based on the optical flow formula. In detail, the operation of PROF can be summarized as the following four steps:

[0068] In step 1, sub-block based affine motion compensation is performed to generate sub-block predictions I(i,j) using the sub-block MVs as derived in the box immediately following paragraph

[0084] above for the 4-parameter affine model and in the box immediately following paragraph

[0087] above for the 6-parameter affine model.

[0069] In step 2, the spatial gradients gx(i,j) and gy(i,j) of each prediction sample are calculated as described in the box immediately following this paragraph.

number

[0070] Further, in step 2, one additional row / column of predicted samples needs to be generated on each side of one sub-block to calculate the gradient. To reduce memory bandwidth and complexity, samples on the extension boundary are copied from the nearest integer pixel position in the reference picture to avoid additional interpolation processes.

[0071] In step 3, the luminance prediction refinement value is calculated as described in the box immediately following this paragraph.

number

[0072] Furthermore, in the current PROF design, after adding a forecast refinement to the original forecast sample, we multiply the value of the refined forecast sample by 1, as described in the box immediately following this paragraph. A clipping operation is performed as a fourth step to clip to within 15 bits.

number

[0073] Since the affine model parameters and pixel positions relative to the subblock center do not change from subblock to subblock, Δv(i,j) can be calculated for the first subblock and reused for other subblocks in the same CU. Let Δx, Δy be the horizontal and vertical offsets from a sample position (i,j) to the center of the subblock to which the sample belongs, and Δv(i,j) can be derived as described in the box immediately following this paragraph.

number

[0074] Based on the affine sub-block MV derivation equations in paragraphs

[0084] and

[0087] above, the MV difference Δv(i,j) can be derived as described in the box immediately following this paragraph.

number

[0075] According to the current LMCS design, chroma residual samples are scaled based on their corresponding luma prediction samples. When newer coding tools are enabled for an inter CU, the luma prediction samples used to scale chroma residual samples through the LMCS in this inter CU are obtained at the end of the sequential application of these newer coding tools.

[0076] 6 is a flow chart showing the workflow of chroma residual scaling in LMCS when DMVR, BDOF and CIIP are all enabled. The outputs from luma L0 prediction 601 and L1 prediction 602 are sequentially fed to DMVR 603 and BDOF 604, and the resulting luma inter prediction 621 along with luma intra prediction 622 are fed to luma intra prediction 605 to averaging 606 to generate an averaged luma prediction 623, which along with chroma residual 608 is fed to chroma residual scaling 607, where chroma residual scaling 607, chroma prediction 610 and chroma reconstruction 609 can work together to generate the final result.

[0077] Current LMCS designs present three challenges to the video decoding process. First, the mapping between different domains requires additional computational complexity and on-chip memory. Second, the fact that the luma and chroma scaling factor derivations use different luma predictors introduces additional complexity. Third, the interaction between LMCS and newer coding tools introduces latency into the decoding process.

[0078] First, in the current LMCS design, both the reconstructed samples in the original domain and the mapped domain are used in various decoding modules. These samples often need to be transformed from one domain to another between different decoding modules, which can lead to both higher computational complexity and more on-chip memory.

[0079] Specifically, for intra, CIIP, and IBC modes, the mapped domain reference samples from the neighboring reconstructed domains of one current CU are used to generate predicted samples. However, for inter modes, motion compensation prediction is performed using the original domain reconstructed samples of the temporal reference picture as reference. The reconstructed samples stored in the DPB are also in the original domain.

[0080] For example, for inter CUs, since the luma reconstruction operation (i.e., addition of predicted samples and residual samples) is performed in the mapped domain, the inter predicted luma samples generated in the original domain need to be transformed to the mapped domain before being used for luma sample reconstruction. In another example, for both intra and inter CUs, inverse (or backward) mapping is always applied to transform the reconstructed luma samples from the mapped domain to the original domain before storing them in the DPB. Such a design not only increases the computational complexity due to the additional forward / backward mapping operation, but also requires more on-chip memory to maintain multiple versions of the reconstructed samples.

[0081] In an actual hardware implementation, the forward and backward (or inverse) mapping functions FwdMap and InvMap may be implemented either using look-up tables (LUTs) or calculated on-the-fly. When a LUT-based solution is used, the possible output elements from the functions FwdMap, InvMap, and cScaleInv may be pre-computed and pre-stored as a LUT, which may then be used for the luma mapping and chroma residual scaling operations of all CUs in the current slice. Assuming the input video is 10-bit, each of the LUTs for FwdMap and InvMap may contain 2 10= 1024 elements, and each element of the LUT is 10 bits. Therefore, the total storage for the LUTs of forward and reverse luma mapping is equal to 2*1024*10 = 20480 bits = 2560 bytes. Meanwhile, to derive the chroma scaling parameter CScaleInv, one 16-entry LUT table cScaleInv needs to be maintained in the encoder and decoder, with each chroma scaling parameter stored in 32 bits. Correspondingly, the memory size used to store the LUT cScaleInv is equal to 16*32 = 512 bits = 64 bytes. The difference between 2560 and 64 indicates the scale of the additional on-chip memory required by the forward and reverse (inverse) mapping operations.

[0082] Furthermore, in newer video coding standards, such as the current VVC, both intra prediction and deblocking filters use reconstructed samples of the neighboring blocks. Therefore, one additional row of reconstructed samples within the width of the current picture / slice needs to be maintained in a buffer, also known as a "line buffer" in video coding. The reconstructed samples in the line buffer are used as references for intra prediction and deblocking operations of at least the CU located in the first row in one CTU. According to the existing LMCS design, the intra prediction and deblocking filters use reconstructed samples in different domains. Therefore, additional on-chip memory is required to store both the original and mapped domain reconstructed samples, which nearly doubles the line buffer size.

[0083] One implementation choice to avoid doubling the line buffer size is to use domain mapping The alternative is to perform the operation on the fly, however this would incur a non-negligible increase in computational complexity.

[0084] Therefore, current designs of LMCS require additional computational complexity and on-chip memory due to the necessary mapping between different domains.

[0085] Second, although both the luma scaling factor derivation method and the chroma scaling factor derivation method in the current design of LMCS use luma predicted sample values ​​to derive corresponding scaling factors, there are differences between their corresponding operations.

[0086] For luma residual scaling, the scaling factor is derived on a sample-by-sample basis by allowing each luma residual sample to have its own scaling factor, however, for chroma residual scaling, the scaling factor is fixed for the entire CU, i.e., all chroma residual samples within a CU share the same scaling factor, which is calculated based on the average of the mapped luma prediction samples.

[0087] In addition, two different LUTs are used to calculate the scaling factors for the luma and chroma residuals: the input to the luma LUT is the mapping model segment index of the original luma prediction sample value, while the input to the chroma LUT is the mapping model segment index of the average value of the mapped luma prediction sample.

[0088] Such differences introduce additional complexity into the encoding process, and a harmonized approach to luma and chroma scaling factor derivation is desirable.

[0089] Third, for the chroma residual scaling in the current design of LMCS, newer coding tools, such as all three modules of DMVR, BDOF, and CIIP, can be called sequentially to generate luma prediction sampling, which is then used to determine the scaling factor of the chroma residual. If the computational complexity of the three modules is large, waiting for their successful completion before performing the chroma residual scaling of LMCS may cause serious latency for the decoding of chroma samples. In the case of affine CUs, the PROF process may have latency issues because each affine CU may execute the PROF process followed by LMCS, which may cause latency issues for the decoding of chroma samples.

[0090] Furthermore, in the current design of LMCS, unnecessary clipping operations are performed during the chroma residual scaling factor derivation process, further increasing the computational complexity and the additional on-chip memory requirements.

[0091] This disclosure aims to solve or mitigate these challenges presented by current designs of LMCS, and more specifically, this disclosure discusses ways in which the complexity of LMCS for hardware codec implementations may be reduced while maintaining coding gain.

[0092] Instead of using the existing LMCS framework, which transforms the predicted / reconstructed samples via a mapping operation, we propose a new method, called Prediction-Dependent Residual Scaling (PDRS), to directly scale the predicted residual without sample mapping. The proposed method can achieve similar effectiveness and coding efficiency as LMCS, but with much smaller implementation complexity.

[0093] In the PDRS procedure, a luma prediction subsample is used to decode the luma residual samples, as shown in Figure 7. A luma prediction sample is obtained (701), a scaling factor is derived using the luma prediction sample (702), the scaling factor is used to scale the luma residual sample (703), and a reconstructed luma sample is calculated by adding the luma prediction sample and the scaled luma residual sample (704).

[0094] Unlike existing LMCS methods that directly transform predicted / reconstructed luma samples into the mapped domain before calculating luma prediction residuals, in the proposed method of PDRS procedure, luma prediction residual samples are derived in the same manner as a normal prediction process in the original domain without a mapping operation, followed by a scaling operation on the luma prediction residuals. The scaling of the luma prediction residuals depends on the corresponding luma prediction sample values ​​and the piecewise linear model. As a result, the forward and luma inverse mapping operations in the current LMCS design can be completely discarded, with all prediction and reconstructed samples involved in the decoding process kept in the original sample domain. Based on the above characteristics, we name the proposed method as prediction-dependent residual scaling.

[0095] Figure 8 is a flow chart showing the workflow of the decoding process when the PDRS procedure is applied in the LMCS process. It shows the elimination of the need for mapping between different domains. Here, except for the residual decoding module (e.g., entropy decoding 801, inverse quantization 802, and inverse transform 803), all other decoding modules (including intra and inter prediction 804, 809, 812, and 816, reconstruction 806 and 813, and all in-loop filters 807 and 814) are operating in the original domain. Specifically, to reconstruct luma samples, the approach proposed in the PDRS procedure only needs to inversely scale luma prediction residual samples Yres to their original amplitude levels, and then add them to luma prediction samples Ypred.

[0096] In the PDRS procedure, the forward and luma sample inverse mapping operations in the existing LMCS design are completely removed. This not only saves / reduces the computational complexity, but also reduces the potential storage size due to the saving of LMCS parameters. For example, when a LUT-based solution is used to implement luma mapping, in the proposed method, the storage previously used to store the two mapping LUTs FwdMap[] and InvMap[] (approximately 2560 bytes) is no longer required. Furthermore, unlike the existing luma mapping method that needs to store the reconstructed luma samples in both the original and mapped domains, the proposed method in the PDRS procedure generates and maintains all predicted and reconstructed samples only in the original domain. Correspondingly, compared to the existing luma mapping, the proposed method in the PDRS procedure can effectively halve the line buffer size used to store the reconstructed samples for intra prediction and deblocking.

[0097] According to one or more embodiments of the PDRS procedure, the luma prediction sample and the luma residual sample are from one and the same location in the luma prediction block and its associated residual block.

[0098] According to one or more embodiments of the PDRS procedure, deriving the scaling factors using the luma prediction samples includes dividing a range of possible luma prediction sample values ​​into a plurality of luma prediction sample segments, calculating one scaling factor for each of the plurality of luma prediction sample segments based on a predetermined piecewise linear model, and determining a scaling factor for the luma prediction sample based on the scaling factors of the plurality of luma prediction sample segments.

[0099] In one example, a luminance prediction sample segment is calculated based on a scaling factor of the luminance prediction sample segments. Determining a scaling factor for the prediction sample includes allocating the luma prediction sample to one segment among a plurality of luma prediction sample segments, and calculating a scaling factor for the luma prediction sample as a scaling factor of the allocated luma prediction sample segment.

[0100] In this example, the plurality of luminance prediction sample segments includes 16 segments in a predetermined 16-section LUT table scaleForward, and the predetermined piecewise linear model for calculating a scaling factor for each of the plurality of luminance prediction sample segments includes 16 values ​​corresponding to the 16 segments in the predetermined LUT table scaleForward.

[0101] In the same example, the scaling factor is calculated based on the assigned luma prediction sample segment, as described in the box immediately following this paragraph.

number

[0102] In the same example, given a luma scaling factor ScaleY, the luma residual sample scaling method can be applied as described in the box immediately following this paragraph.

number

[0103] The motivation behind this example is that the forward mapping in current LMCS is based on a piecewise linear model. Both the original luma samples and the luma predicted samples are in the same piecewise linear model. That is, if the two luma samples and the predicted luma samples are located in the same segment defined by two pivot points InputPivot[i] and InputPivot[i+1], the two forward mapping functions of the original luma samples and the predicted luma samples are exactly the same. Correspondingly, Y'res=FwdMap(Yorg)-FwdMap(Ypred)=FwdMap(Yorg-Ypred)==FwdMap(Yres). By applying inverse mapping to both sides of this equation, the corresponding decoder-side reconstruction operation can be expressed as Yrecon=Ypred+InvMap(Y'res).

[0104] In other words, in the situation where both the original luma samples and the luma prediction samples are located in the same partition, this can be achieved through one residual scaling operation in the decoding process, as the luma mapping method in LMCS is implemented in this possible implementation.

[0105] Although such a conclusion is derived based on the assumption that both the original luma samples and the luma prediction samples are located within the same segment defined by the two pivot points InputPivot[i] and InputPivot[i+1], this possible implementation of this example can still be used as a simplification and / or approximation for the existing luma mapping operation in VVC even when the original luma samples and the luma prediction samples are located within different segments of the piecewise linear model. Experimental results show that such simplifications and / or approximations have little impact on the coding performance.

[0106] Again, this example is based on the assumption that both the original and predicted luma sample values ​​are located within the same segment in the piecewise linear mode. In this case, the forward / backward mapping functions applied to the original and predicted luma samples are the same, and therefore it is safe to simply rely on the luma prediction samples to calculate the corresponding residual scaling factors.

[0107] However, when the predicted samples of a CU are not sufficiently accurate (e.g., in the case of intra-predicted CUs where samples far away from the reference samples are usually predicted less accurately), the predicted samples and the original samples often lie in different segments of the piecewise linear model. In this case, the scaling factor derived based on the predicted sample values ​​may not be reliable in reflecting the original mapping relationship between the residual samples in the original (i.e., unmapped) domain and the residual samples in the mapped domain.

[0108] FIG. 9 is an illustration of residual mapping error caused by simply using predicted samples to derive scaling factors. In FIG. 9, the triangular solid points represent pivot control points of various segments in the piecewise linear function, the circular solid points represent original sample values ​​and predicted sample values, Yorg and Ypred are original samples and predicted samples in the original (i.e., unmapped) domain, and Y'org and Y'pred are mapped samples of Yorg and Ypred, respectively. Yres and Y'res are corresponding residuals in the original domain and the mapped domain when the existing sample-based luma mapping method in VVC is applied, and Y'resScale is the mapped residual sample derived based on the proposed luma residual scaling scheme. As shown in FIG. 9, the original samples and predicted samples are not in the same interval of the piecewise linear model, so the scaling factor derived based on the predicted sample may not be accurate enough to generate a scaled residual (i.e., Y'resScale) that approximates the original residual in the mapped domain (i.e., Y'res).

[0109] In the second example, both the original and predicted luminance sample values ​​are in a piecewise linear mode. There is no need to assume that the two nodes are located in the same segment.

[0110] In this second example, instead of deriving the scaling factor directly from the segment of the piecewise linear model in which the luminance prediction sample is located, the scaling factor is calculated as the average of the scaling factors of N adjacent segments, where N is a positive integer.

[0111] In this second example, determining a scaling factor for the luma prediction sample based on the scaling factors of the multiple luma prediction sample segments includes allocating the luma prediction sample to one of the multiple luma prediction sample segments and calculating the scaling factor for the luma prediction sample as an average of scaling factors of several luma prediction sample segments adjacent to the allocated luma prediction sample segment.

[0112] More specifically, in one possible implementation of this second example, the scaling factor may be calculated based on the allocated luma prediction sample segments, as described in the following steps. 1) Find or obtain the segment index IdxY to which the piecewise linear model to which PredY belongs in the original domain corresponds. 2) If Y'res >= 0, the luminance residual scaling factor is calculated as follows:

number

number

[0113] In a second possible implementation of this second example, which is otherwise identical to the implementation described above, the scaling factor may be calculated based on the assigned luma prediction sample segment, as described in the box immediately following this paragraph:

number

[0114] The above two possible implementations of this second example differ only in the selection of the N luminance prediction sample domain value segments based on the assigned segments.

[0115] In one chroma sample reconstruction procedure, as shown in FIG. 10, a luma prediction sample value is obtained to decode both a luma residual sample and a chroma residual sample at an input position (1001), then a luma prediction sample associated with the luma residual sample is obtained (1002), then a chroma prediction sample associated with the chroma residual sample is obtained (1003), the luma prediction sample is used to derive a first scaling factor for the luma residual sample and a second scaling factor for the chroma residual sample (1004), the first scaling factor is used to scale the luma residual sample (1005), the second scaling factor is used to scale the chroma residual sample (1006), a reconstructed luma sample is calculated by adding the luma prediction sample and the scaled luma residual sample (1007), and a reconstructed chroma sample is calculated by adding the chroma prediction sample and the scaled chroma residual sample (1008).

[0116] The chroma sample reconstruction procedure aims to harmonize the scaling methods of the luma and chroma residuals to achieve a more unified design.

[0117] According to one or more embodiments of the chroma sample reconstruction procedure, the luma prediction sample value is the average of all luma prediction samples in the coding unit (CU) that contains the input position. In these embodiments, a chroma scaling derivation method is used to calculate a scaling factor for the luma residual, and more specifically, instead of deriving one scaling factor separately for each luma residual sample, one shared scaling factor calculated based on the average of the luma prediction samples is used to scale the luma residual samples of the entire CU.

[0118] According to another embodiment of the chroma sample reconstruction procedure, the luma prediction sample value is the average of all luma prediction samples in a predefined sub-block sub-divided from the coding unit (CU) containing the input position. In this embodiment, one CU is first equally divided into a number of M×N sub-blocks, and then, for each sub-block, all or a portion of the luma prediction samples are used to derive a corresponding scaling factor that is used to scale both the luma and chroma residuals of the sub-block. Compared with the first method, the second method can reduce the correlation of the luma prediction samples outside the sub-block. Since the first method is excluded from calculating the scaling factor of the subblock, the spatial accuracy of the estimated scaling factor can be improved. Meanwhile, the second method can also reduce the latency of the luma and chroma residual reconstruction, on the premise that the scaling of the luma and chroma residual in one subblock can be started immediately after the luma prediction of the subblock is finished, i.e., without waiting for the complete generation of the luma prediction samples of the entire CU.

[0119] According to a third embodiment of the chroma sample reconstruction procedure, the luma prediction sample domain values ​​include collocated luma prediction samples. In this embodiment, the luma residual scaling method is extended to scale the chroma residuals, and a different scaling factor for each chroma residual sample is derived based on its collocated luma prediction sample value.

[0120] In the above embodiment of the chroma sample reconstruction procedure, it is proposed to use the same LUT used to compute the luma scaling factor for scaling the chroma residual. In one example, to derive the CU level scaling factor ScaleC for the chroma residual, it can be done as follows: 1) Calculate the average of the luminance prediction samples (represented in the original domain) within the CU, denoted as avgY. 2) Find or get the corresponding segment index IdxY of the piecewise linear model to which avgY belongs. 3) Calculate the value of ScaleC as follows: ScaleC=scaleForward[IdxY] where scaleForward[i], i=0...15 is a predefined 16-way LUT, It is calculated as: scaleForward[i]=(OrgCW< <SCALE_FP_PREC) / SignaledCW[i] where OrgCW and SignaledCW[i] are the numbers of codewords of the i-th segment in the original and mapped domains, and SCALE_FP is the precision of the scaling factor.

[0121] The above example can be easily extended if a scaling factor for the chroma residual is derived for each sub-block of the current CU, in which case in the first step above avgY is calculated as the average of the luma prediction samples in the original domain of the sub-block, while steps 2 and 3 remain the same.

[0122] In the second chroma sample reconstruction procedure, as shown in FIG. 11, during a luma prediction process for a coding unit (CU), multiple luma prediction samples are obtained (1101) by skipping some of the predetermined intermediate luma prediction stages, the obtained multiple luma prediction samples are used to derive scaling factors for chroma residual samples in the CU (1102), the scaling factors are used to scale the chroma residual samples in the CU (1103), and the reconstructed chroma samples are calculated by adding the chroma prediction samples and the scaled chroma residual samples in the CU (1104).

[0123] According to one or more embodiments of the second chroma sample reconstruction procedure, the predetermined intermediate luma prediction stage may include a decoder-side motion vector derivation (DMVR), a BD The bi-directional prediction module includes one or more of the following: Bi-Directional Optical Flow (OF), Combined Inter and Intra Prediction (CIIP), and Inter-Prediction samples derived before the DMVR, BDOF / PROF, and CIIP Intra / Inter combined processes are used to derive a scaling factor for the chroma residual.

[0124] 12 is a flowchart showing the workflow of the LMCS decoding process in one example of this embodiment of the second chroma sample reconstruction procedure, in which DMVR, BDOF, and CIIP are not applied to generate luma prediction samples for chroma scaling. Here, instead of waiting for the DMVR 1203, BDOF 1204, and / or luma intra prediction unit 1205 of CIIP to be fully finished, the chroma residual scaling process 1208 can start as soon as prediction samples 1221 and 1222 based on the initial L0 and L1 luma predictions 1201 and 1202 are available.

[0125] In FIG. 12, in addition to the original averaging operation 1206, an additional averaging operation 1211 is required to combine the initial L0 predicted samples 1221 and L1 predicted samples 1222 before the DMVR 1203, BDOF 1204, and / or CIIP 1205.

[0126] To reduce complexity, in a second example of this embodiment of the second chroma sample reconstruction procedure, the first L0 prediction sample may be used to derive a scaling factor for the chroma residual.

[0127] 13 is a flowchart showing the workflow of the LMCS decoding process in a second example of this embodiment of the second chroma sample reconstruction procedure, in which an initial uni-prediction signal is applied to generate luma prediction samples for chroma scaling. No additional averaging operations are required in addition to the original averaging operation 1306. The initial L0 prediction samples 1321 are used to derive a scaling factor for the chroma residual before DMVR 1303, BDOF 1304, and / or CIIP 1305.

[0128] In a third example of this embodiment of the second chroma sample reconstruction procedure, one initial prediction signal (L0 or L1) is adaptively selected as the luma prediction sample used to derive the chroma residual scaling factor. In one possible implementation of this example, among the initial prediction signals (L0 or L1), the one whose reference picture has a smaller picture order count (POC) distance to the current picture is selected to derive the chroma residual scaling factor.

[0129] Another embodiment of the second chroma sample reconstruction procedure proposes to disable only DMVR, BDOF / PROF while enabling CIIP to generate inter-prediction samples used to determine chroma residual scaling factors. Specifically, in this method, the inter-prediction samples derived before DMVR and BDOF / PROF are averaged, then combined with the intra-prediction samples of CIIP, and finally the combined prediction samples are used as prediction samples to determine chroma residual scaling factors.

[0130] In yet another embodiment of the second chroma sample reconstruction procedure, it is proposed to disable only BDOF / PROF while maintaining DMVR and CIIP for generating the predicted samples used to determine the chroma residual scaling factor.

[0131] In yet another embodiment of the second chroma sample reconstruction procedure, it is proposed to maintain BDOF / PROF and CIIP while disabling DMVR when deriving the luma prediction samples used to determine the chroma residual scaling factor.

[0132] Furthermore, although the methods in the above embodiment of the second chroma sample reconstruction procedure are shown as being designed to reduce the latency of chroma prediction residual scaling, it should be noted that these methods may also be used to reduce the latency of luma prediction residual scaling. For example, all of these methods can also be applied to the PDRS method described in “Luminance Mapping Based on Prediction-Dependent Residual Scaling.”

[0133] According to the existing DMVR design, in order to reduce the computational complexity, the prediction samples used for DMVR motion refinement are generated using a 2-tap bilinear filter instead of the default 8-tap interpolation. After the refined motion is determined, the default 8-tap filter is applied to generate the final prediction sample of the current CU. Therefore, in order to reduce the chroma residual decoding latency caused by DMVR, it is proposed to use the luma prediction sample generated by the bilinear filter (the average of the L0 and L1 prediction samples if the current CU is bi-predicted) to determine the scaling factor of the chroma residual.

[0134] According to one chroma residual sample reconstruction procedure, one or more chroma residual samples are extracted from the output of the bilinear filter of the Decoder-side Motion Vector Derivation (DMVR), as shown in FIG. A luma prediction sample value is selected (1401), one or more selected luma prediction sample values ​​are scaled to another plurality of luma prediction sample values ​​having the same bit depth as the original encoding bit depth of the input video (1402), the luma prediction sample value having the same bit depth as the original encoding bit depth of the input video is used to derive a scaling factor for decoding one or more chroma residual samples (1403), the scaling factor is used to scale the one or more chroma residual samples (1404), and the one or more chroma residual samples are reconstructed by adding the one or more scaled chroma residual samples and their corresponding chroma prediction samples (1405).

[0135] In one or more embodiments of the chroma residual sample reconstruction procedure, selecting one or more luma prediction sample values ​​from the output of the bilinear filter of the DMVR includes selecting L0 and L1 luma prediction samples from the output of the bilinear filter of the DMVR.

[0136] 15 is a flow chart illustrating the workflow of the LMCS decoding process in one such embodiment of a chroma residual sample reconstruction procedure. The L0 and L1 prediction samples 1521 and 1522 from the output of the bilinear filter 1512 component of the DMVR 1503 are fed to an average 1511 to derive a chroma residual scaling input 1523 to be used in chroma residual scaling 1507 to decode one or more chroma residual samples.

[0137] In these embodiments, there is a problem of bit code depth. In order to save the internal storage size used by the DMVR, the intermediate L0 and L1 prediction samples generated by the bilinear filter of the DMVR are 10-bit precision. This is different from the representation bit depth of the immediate prediction samples of normal bi-prediction, which is equal to 14 bits. Therefore, the intermediate prediction samples output from the bilinear filter cannot be directly applied to determine the chroma residual scaling factor due to their different precision.

[0138] To address this issue, it is first proposed to align the DMVR intermediate bit depth with the intermediate bi-depth used for regular motion compensation interpolation, i.e., increase the bit depth from 10 bits to 14 bits. Then, the existing averaging process applied to generate the regular bi-predicted signal can be reused to generate the corresponding predicted samples for the determination of the chroma residual scaling factor.

[0139] In one example of these embodiments, the one or more selected luma prediction sample values ​​are combined with another plurality of luma prediction sample values ​​having the same bit depth as the original coding bit depth of the input video. Adjusting to the input video includes increasing an internal bit depth of the L0 and L1 luma prediction samples from the output of the bilinear filter of the DMVR to 14 bits through a left shift, obtaining a 14-bit average luma prediction sample value by averaging the 14-bit shifted L0 and L1 luma prediction sample values, and converting the internal bit depth of the 14-bit average luma prediction sample value to the original coding bit depth of the input video through a right shift.

[0140] More specifically, in this example, the saturation scaling factor is determined by the steps described in the box immediately following this paragraph.

number

[0141] In another embodiment of the chroma residual sampling reconstruction procedure, selecting one or more luma prediction sampling values ​​from the output of the bilinear filter of the DMVR and adjusting the one or more selected luma prediction sampling values ​​to another plurality of luma prediction sampling values ​​at a bit depth same as the original encoding bit depth of the input video includes selecting one luma prediction sampling from the L0 and L1 luma prediction samplings from the output of the bilinear filter of the DMVR, adjusting the one selected luma prediction sampling by changing an internal bit depth of the one selected luma prediction value to the original encoding bit depth of the input video through shifting, and using the adjusted luma prediction sampling at the bit depth same as the original encoding bit depth of the input video as the luma prediction sampling.

[0142] FIG. 16 is a flowchart showing the workflow of the LMCS decoding process in one such alternative embodiment of the chroma residual sample reconstruction procedure. The L0 prediction samples 1621 from the output of the linear filter 1612 component are used in chroma residual scaling 1607 to decode one or more chroma residual samples.

[0143] In one example of one such alternative embodiment of the chroma residual sample reconstruction procedure, the chroma scaling factor is determined by shifting the luma samples output from the bilinear filter to the original coding bit depth of the input video, as described in the box immediately following this paragraph.

number

[0144] According to the second chroma residual sample reconstruction procedure, as shown in FIG. 17, one or more luma reference sample values ​​are selected from a reference picture (1701), the one or more selected luma reference sample values ​​are converted to luma sample values ​​(1702), the converted luma sample values ​​are used to derive a scaling factor (1703), the scaling factor is used to scale one or more chroma residual samples (1704), and one or more chroma residual samples are reconstructed by adding one or more scaled chroma residual samples and their corresponding chroma prediction samples (1705).

[0145] In one or more embodiments of the second chroma residual sample reconstruction procedure, selecting one or more luma reference sample values ​​from the reference picture and converting the one or more selected luma reference sample values ​​to luma sample values ​​includes obtaining both L0 and L1 luma reference sample values ​​from the L0 and L1 reference pictures and averaging the L0 and L1 luma reference sample values ​​as the converted luma sample value.

[0146] In another embodiment of the second chroma residual sample reconstruction procedure, selecting one or more luma reference samples from the reference picture and converting the one or more selected luma reference samples to luma sample values ​​includes selecting one luma reference sample value from the L0 and L1 luma reference sample values ​​from the L0 and L1 reference pictures and using the one selected luma reference sample value as the converted luma sample value.

[0147] According to the existing LMCS design, the reconstructed luminance samples adjacent to the 64×64 region where the current CU is located are used to calculate the chroma residual scaling factor of the CU within the region. Further, one clipping operation, i.e., Clip1(), is applied to clip the luminance adjacent samples reconstructed to the dynamic range of the internal bit depth (i.e., in the range of [0, (1<<bitDepth)-1]) before the average is calculated.

[0148] Specifically, this method first fetches 64 left adjacent luminance samples and 64 upper adjacent luminance samples of the corresponding 64×64 region to which the current CU belongs, then calculates the average of the left adjacent samples and the upper adjacent samples, i.e., avgY, and finds the segment index YIdx of avgY in the LMCS piecewise linear model, and finally derives the chroma residual CScale Inv = cScaleInv[YIdx].

[0149] Specifically, in the current VVC draft, the method for deriving the corresponding average luminance is as follows. The application of the Clip1() operation is shown in prominent font face and size. :

Number

[0150] However, in the reconstruction process, after adding the predicted samples to the residual samples of one CU, the resulting sample values are already clipped to the dynamic range of the internal bit depth. This means that all adjacent reconstructed luminance samples around the current 64×64 region are guaranteed to be within the range of the internal bit depth. Therefore, their average, i.e., avgY, cannot exceed this range either. As a result, the existing clipping (i.e., Clip1()) is for the corresponding chroma residual scale To further reduce the complexity and memory requirements of the LMCS design, it is proposed to eliminate the clipping operation when averaging neighboring reconstructed luma samples to derive the chroma residual scaling factor.

[0151] According to a first aspect of the present application, as shown in FIG. 18, a plurality of reconstructed luma samples from a first predetermined region adjacent to a second predetermined region in which the CU is located are selected during decoding of the CU (1801), an average of the plurality of reconstructed luma samples is calculated (1802), and the average of the plurality of reconstructed luma samples is directly used (1803) without clipping when deriving a chroma residual scaling factor for decoding the CU.

[0152] In one or more embodiments of the first aspect of the present application, the average of the multiple reconstructed luminance samples is an arithmetic mean of the multiple reconstructed luminance samples.

[0153] In one or more embodiments of the first aspect of the present application, directly using an average of multiple reconstructed luma samples without clipping when deriving a chroma residual scaling factor for decoding the CU includes identifying a segment index for the average in a predetermined piecewise linear model, and deriving a luma residual scaling factor for decoding the CU based on a gradient of the linear model of the segment.

[0154] In one or more embodiments of a first aspect of the present application, a plurality of reconstructed luma samples within a first predetermined region are generated by generating luma prediction samples and luma residual samples within the first predetermined region, adding the luma residual samples to the luma prediction samples, and clipping the added luma samples to a dynamic range of an encoding bit depth.

[0155] In one or more embodiments of the first aspect of the present application, the plurality of reconstructed luma samples are a plurality of forward-mapped inter-luma reconstructed samples.

[0156] In one or more embodiments of the first aspect of the present application, the second predetermined region is a 64×64 region in which the CU is located.

[0157] 19, the first predetermined region may include top-neighboring samples in a 1×64 region 1902 immediately above the second predetermined region 1904. Alternatively or additionally, the first predetermined region may include left-neighboring samples in a 64×1 region 1903 immediately to the left of the second predetermined region 1904.

[0158] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another, for example according to a communication protocol. As such, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for execution of the implementations described herein. A computer program product may include a computer-readable medium.

[0159] Furthermore, the above method can be implemented using ASICs (application specific integrated circuits), D The methods may be implemented using an apparatus including one or more circuits, including digital signal processors (SPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components. The apparatus may use the circuits in combination with other hardware or software components to perform the methods described above. Each module, sub-module, unit, or sub-unit disclosed above may be implemented at least in part using one or more circuits.

[0160] The present disclosure may include dedicated hardware implementations such as application specific integrated circuits, programmable logic arrays, and other hardware devices. The hardware implementations may be configured to implement one or more of the methods described herein. Examples that may include devices and systems of various implementations may broadly include various electronic and computing systems. One or more examples described herein may implement functions using two or more specific interconnected hardware modules, or devices, with associated control and data signals that may communicate between, through, or as part of an application specific integrated circuit. Thus, the disclosed devices or systems may encompass software, firmware, and hardware implementations. Terms such as "module," "sub-module," "circuit," "sub-circuit," "unit," or "sub-unit" may include memory (shared, dedicated, or group) that stores code or instructions that may be executed by one or more processors. A module as used herein may include one or more circuits, with or without stored code or instructions. A module, or circuit, may include one or more components that are connected.

[0161] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure in accordance with its general principles, such departures from the disclosure being within known or customary practice in the art. The specification and embodiments are intended to be merely exemplary, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0162] It will be understood that the present disclosure is not limited to the precise examples described above and illustrated in the accompanying drawings, but various modifications and changes can be made without departing from the scope thereof, which is intended to be limited only by the scope of the appended claims.

[0163] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 001,229, filed March 27, 2020. The entire disclosure of the aforementioned application is incorporated herein by reference in its entirety for all purposes.

Claims

1. Selecting a plurality of reconstructed luma samples from a first predetermined region adjacent to a second predetermined region in which the coding unit is located; averaging a plurality of said reconstructed luminance samples; deriving a chroma residual scaling factor directly using an average of a plurality of said reconstructed luma samples without clipping; Encoding LMCS (luma mapping with chroma scaling) related information into the bitstream; Video encoding methods.

2. The video encoding method of claim 1 , wherein the average of a plurality of the reconstructed luma samples is an arithmetic mean of a plurality of the reconstructed luma samples.

3. Deriving a chroma residual scaling factor directly using an average of a plurality of the reconstructed luma samples without clipping, comprising: Identifying an average segment index in a predetermined piecewise linear model; deriving a luminance residual scaling factor based on the segment index; The video encoding method of claim 1 , comprising:

4. The plurality of reconstructed luma samples include generating luma prediction samples and luma residual samples; adding the luma residual samples to the luma prediction samples; clipping the luma prediction samples to a dynamic range of an encoding bit depth; It is generated by 2. The video encoding method of claim 1.

5. The video encoding method of claim 1 , wherein the reconstructed luma samples are derived based on forward-mapped inter-predicted samples.

6. The video encoding method of claim 1 , wherein the second predetermined region is a 64×64 region in which the coding unit is located.

7. 7. The video encoding method of claim 6, wherein the first predetermined region includes a 1x64 region immediately above the second predetermined region and a 64x1 region immediately to the left of the second predetermined region.

8. one or more processors; a non-transitory storage device coupled to one or more of said processors; A plurality of programs, When executed by one or more of said processors, the method causes a computing device to perform the method of any one of claims 1 to 7 and stores the generated bitstream in said non-transitory storage device. A plurality of programs stored in the non-transitory storage device; 2. A computing device comprising:

9. When executed by a computing device having one or more processors and a non-transitory storage device coupled to the one or more processors, the method causes the one or more processors to perform the method of any one of claims 1 to 7, generating a bitstream, and storing the generated bitstream in the non-transitory storage device. program.

10. 1. A method for storing a bitstream in a non-transitory storage device, comprising: The bitstream is generated by an encoding method, The encoding method comprises: Selecting a plurality of reconstructed luma samples from a first predetermined region adjacent to a second predetermined region in which the coding unit is located; averaging a plurality of said reconstructed luminance samples; deriving a chroma residual scaling factor directly using an average of a plurality of said reconstructed luma samples without clipping; Encoding LMCS (luma mapping with chroma scaling) related information into the bitstream; Including, A method for storing a bitstream in a non-transitory storage device.

Images