Apparatus and method for bit depth control for bidirectional optical flow

Abstract

Methods, apparatuses, and non-transitory computer-readable storage media for decoding a video signal are provided. The method includes obtaining a first reference picture I (0) and a second reference picture I (1) , obtaining first prediction samples I (0) (i,j) of the video block from a reference block in the first reference picture I (0) , obtaining second prediction samples I (1) (i,j) of the video block from a reference block in the second reference picture I (1) , controlling an internal bit-depth of the BDOF by applying a right shift to an internal BDOF parameter, and obtaining final bi-prediction samples of the video block based on the BDOF being applied to the video block based on the first prediction samples I (0) (i,j) and the second prediction samples I (1) (i,j).

Description

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Provisional Application No. 62 / 866,607, filed on June 25, 2019 and Provisional Application No. 62 / 867,185, filed on June 26, 2019, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure relates to video encoding / decoding and compression. More specifically, this disclosure relates to methods and apparatus for a bidirectional optical flow (BDOF) method for video encoding / decoding. Background Technology

[0004] Various video codec techniques can be used to compress video data. Video codecs are performed according to one or more video codec standards. Examples of video codec standards include Universal Video Codec (VVC), Joint Explore Test Model (JEM), High Efficiency Video Codec (H.265 / HEVC), High-Level Video Codec (H.264 / AVC), and Moving Picture Experts Group (MPEG) codecs. Video codecs typically use prediction methods that utilize redundancy present in video images or sequences (e.g., inter-frame prediction, intra-frame prediction, etc.). A key goal of video codec techniques is to compress video data to a lower bitrate while avoiding or minimizing video quality degradation. Summary of the Invention

[0005] Examples of this disclosure provide methods and apparatus for motion vector prediction in video encoding and decoding.

[0006] According to a first aspect of this disclosure, a method for decoding a video signal is provided. The method may include obtaining a first reference image I associated with a video block at a decoder. (0) Second reference image I (1) In the order of display, the first reference image I (0) Before the current image and the second reference image I (1) Following the current image, the method may further include, at the decoder, retrieving from the first reference image I. (0) The reference block in the video block obtains the first prediction sample I of the video block. (0) (i, j). i and j can represent the coordinates of a sample point within the current image. The method may include retrieving data from a second reference image I at the decoder. (1) The reference block in the video block obtains the second prediction sample I of the video block. (1)(i,j). The method can include controlling, at the decoder, an internal bit depth of the BDOF by applying a right shift to an internal BDOF parameter. The BDOF is independent of an input video bit depth. The internal BDOF parameter can include a horizontal gradient value and a vertical gradient value derived based on a first prediction sample I (0) (i,j), a horizontal gradient value and a vertical gradient value derived based on a second prediction sample I (1) (i,j), and a sample difference value between the first prediction sample I (0) (i,j) and the second prediction sample I (1) (i,j). The method can include applying, at the decoder, the BDOF based on the first prediction sample I (0) (i,j) and the second prediction sample I (1) (i,j) to the video block to obtain final bi-prediction samples of the video block.

[0007] According to a second aspect of the disclosure, a method of decoding a video signal is provided. The method can include obtaining, at a decoder, a first reference picture I (0) and a second reference picture I (1) associated with a video block. In display order, the first reference picture I (0) is before a current picture and the second reference picture I (1) is after the current picture. The method can further include obtaining, at the decoder, a first prediction sample I (0) (i,j) of the video block from a reference block in the first reference picture I (0) (i,j). i and j represent coordinates of a sample within the current picture. The method can include obtaining, at the decoder, a second prediction sample I (1) (i,j) of the video block from a reference block in the second reference picture I (1) (i,j). The method can further include controlling, at the decoder and when an internal bit depth is greater than 12 bits, an internal bit depth of the BDOF by applying a right shift to an internal BDOF parameter to align precision of an output prediction signal to a constant. The internal BDOF parameter can include a horizontal gradient value and a vertical gradient value derived based on the first prediction sample I (0) (i,j), a horizontal gradient value and a vertical gradient value derived based on the second prediction sample I (1) (i,j), and a sample difference value between the first prediction sample I (0) (i,j) and the second prediction sample I (1) (i,j). The method can include applying, at the decoder, the BDOF based on the first prediction sample I (0) (i,j) and the second prediction sample I (1)(i,j) are applied to the video block to obtain final bi-predicted samples of the video block. The method can further include obtaining, at the decoder, the output prediction signal based on the final bi-predicted samples.

[0008] According to a third aspect of the disclosure, a computing device for decoding a video signal is provided. The computing device can include one or more processors, a non-transitory computer-readable memory storing instructions executable by the one or more processors. The one or more processors can be configured to obtain, at a decoder, a first reference picture I (0) and a second reference picture I (1) picture, the first reference picture I (0) preceding the current picture and the second reference picture I 1) following the current picture, the one or more processors can be further configured to obtain, at the decoder, first prediction samples I (0) (i,j) of the video block from a reference block in the first reference picture I (0) (i,j). i and j represent coordinates of one sample within the current picture. The one or more processors can be configured to obtain, at the decoder, second prediction samples I (1) (i,j) of the video block from a reference block in the second reference picture I (1) (i,j). The one or more processors can be further configured to control, at the decoder, an internal bit-depth of bi-directional optical flow (BDOF) by applying a right shift to internal BDOF parameters. The BDOF is independent of an input video bit-depth. The internal BDOF parameters include horizontal gradient values and vertical gradient values derived based on the first prediction samples I (0) (i,j), horizontal gradient values and vertical gradient values derived based on the second prediction samples I (1) (i,j), and sample difference values between the first prediction samples I (0) (i,j) and the second prediction samples I (1) (i,j); and obtain, at the decoder, final bi-predicted samples of the video block based on the BDOF based on the first prediction samples I (0) (i,j) and the second prediction samples I (1) (i,j).

[0009] According to a fourth aspect of the disclosure, a non-transitory computer-readable storage medium storing instructions is provided. When the instructions are executed by one or more processors of the apparatus, the instructions can cause the apparatus to perform the following steps: obtaining, at a decoder, a first reference picture I (0) and a second reference picture I (1). In display order, a first reference picture I (0) preceding the current picture and a second reference picture I (1) following the current picture. The instructions can further cause the device to perform the step of obtaining, at the decoder, first prediction samples I (0) (i,j) of the video block from a reference block in the first reference picture I (0) (i,j). i and j represent coordinates of one sample within the current picture. The instructions can additionally further cause the device to perform the step of obtaining, at the decoder, second prediction samples I (1) (i,j) of the video block from a reference block in the second reference picture I (1) (i,j). The instructions can further cause the device to perform the step of controlling, at the decoder and when an internal bit depth is greater than 12 bits, the internal bit depth of an internal bi-directional optical flow (BDOF) by applying a right shift to the BDOF parameters to align precision of an output prediction signal to a constant. The internal BDOF parameters include horizontal gradient values and vertical gradient values derived based on the first prediction samples I (0) (i,j), horizontal gradient values and vertical gradient values derived based on the second prediction samples I (1) (i,j), and sample difference values between the first prediction samples I (0) (i,j) and the second prediction samples I (1) (i,j). The instructions can additionally further cause the device to perform the step of obtaining, at the decoder, final bi-prediction samples of the video block based on the BDOF from the first prediction samples I (0) (i,j) and the second prediction samples I (1) (i,j) being applied to the video block. The instructions can further cause the device to perform the step of obtaining, at the decoder, the output prediction signal based on the final bi-prediction samples. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which are incorporated in and form a part of the specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0011] Figure 1 is a block diagram of an encoder according to an example of the present disclosure.

[0012] Figure 2 is a block diagram of a decoder according to an example of the present disclosure.

[0013] Figure 3A is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0014] Figure 3Bis a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0015] Figure 3C is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0016] Figure 3D is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0017] Figure 3E is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.

[0018] Figure 4 is a diagram of a bi-directional optical flow (BDOF) model according to an example of the present disclosure.

[0019] Figure 5 is a bit depth control method for BDOF according to an example of the present disclosure.

[0020] Figure 6 is a bit depth control method for BDOF according to an example of the present disclosure.

[0021] Figure 7 is a diagram illustrating a computing environment coupled with a user interface according to an example of the present disclosure. DETAILED DESCRIPTION

[0022] Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers represent the same or similar elements between the several figures. The implementations set forth in the following description of exemplary embodiments do not represent all of the implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with aspects related to the present disclosure as claimed in the appended claims.

[0023] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0024] It should be understood that, although terms such as“first,”“second,”“third,” etc. can be used herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one category of information from another category of information. For example, without departing from the scope of the present disclosure, first information can be referred to as second information; and similarly, second information can also be referred to as first information. As used herein, the term“if’ can be understood to mean“when” or“in response to a determination” or“in response to a judgment” depending on the context.

[0025] The first version of the HEVC standard was finalized in October 2013, providing approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard, H.264 / MPEG AVC. While the HEVC standard provides more significant coding improvements than its predecessor, evidence suggests that coding efficiency superior to HEVC can be achieved with additional coding tools. On this basis, both VCEG and MPEG have begun exploratory work on new coding technologies for future video coding standardization. ITU-T VECG and ISO / IEC MPEG formed the Joint Video Exploration Team (JVET) in October 2015 to begin important studies of advanced techniques that can achieve substantial enhancements to coding efficiency. A reference software, referred to as the Joint Exploration Model (JEM), is maintained by the JVET by integrating several additional coding tools on top of the HEVC Test Model (HM).

[0026] In October 2017, a Call for Proposals (CfP) on video compression with capabilities beyond HEVC was issued by ITU-T and ISO / IEC. In April 2018, 23 CfP responses were received and evaluated at the 10th JVET meeting, demonstrating compression efficiency gains of approximately 40% beyond HEVC. Based on such evaluation results, the JVET launched a new project to develop a new generation of video coding standard, named Versatile Video Coding (VVC). In the same month, a reference software codebase, called VVC Test Model (VTM), was established to demonstrate a reference implementation of the VVC standard.

[0027] Like HEVC, VVC is built on a block-based hybrid video coding framework.

[0028] Figure 1 A general diagram of a block-based video encoder for VVC is shown. Specifically, Figure 1A typical encoder 100 is shown. The encoder 100 has a video input 110, a motion compensation 112, a motion estimation 114, an intra / inter mode decision 116, a block predictor 140, an adder 128, a transform 130, a quantization 132, prediction related information 142, an intra prediction 118, a picture buffer 120, a dequantization 134, an inverse transform 136, an adder 126, a memory 124, an in-loop filter 122, an entropy coding 138, and a bitstream 144.

[0029] In the encoder 100, a video frame is partitioned into video blocks for processing. For each given video block, a prediction is formed based on either an inter prediction method or an intra prediction method.

[0030] The prediction residual representing the difference between the current video block (part of the video input 110) and the predictor of the current video block (part of the block predictor 140) is sent from the adder 128 to the transform 130. The transform coefficients are then sent from the transform 130 to the quantization 132 for entropy reduction. The quantized coefficients are then fed to the entropy coding 138 to generate the compressed video bitstream. As Figure 1 As shown in the middle, the prediction related information 142 (such as video block partition information, motion vectors (MVs), reference picture indices, and intra prediction modes) from the intra / inter mode decision 116 is also fed through the entropy coding 138 and saved into the compressed bitstream 144. The compressed bitstream 144 comprises the video bitstream.

[0031] In the encoder 100, decoder related circuitry is also needed for the purpose of prediction in order to reconstruct the pixels. First, the prediction residual is reconstructed by the dequantization 134 and the inverse transform 136. This reconstructed prediction residual is combined with the block predictor 140 to generate the unfiltered reconstructed pixels of the current video block.

[0032] Spatial prediction (or "intra prediction") uses pixels from already coded neighboring blocks in the same video frame (which are referred to as reference samples) to predict the current video block.

[0033] Temporal prediction (also referred to as "inter prediction") uses reconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given coding unit (CU) or coding block is typically signaled by one or more MVs, where the one or more MVs indicate the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index is additionally signaled, where the reference picture index is used to identify from which reference picture in the reference picture store the temporal prediction signal comes.

[0034] Motion estimation 114 accesses video input 110 and a signal from picture buffer 120 and outputs a motion estimation signal to motion compensation 112. Motion compensation 112 accesses video input 110, a signal from picture buffer 120, and a motion estimation signal from motion estimation 114 and outputs a motion compensated signal to intra / inter mode decision 116.

[0035] After performing spatial and / or temporal prediction, intra / inter mode decision 116 in encoder 100 selects the best prediction mode, e.g., based on rate-distortion optimization methods. The block prediction value 140 is then subtracted from the current video block, and the resulting prediction residual is de-correlated using transform 130 and quantization 132. The resulting quantized residual coefficients are de-quantized by inverse quantization 134 and inverse transformed by inverse transform 136 to form a reconstructed residual, which is then added back to the prediction block to form a reconstructed signal for the CU. Further in-loop filters 122, such as a deblocking filter, sample adaptive offset (SAO), and / or adaptive loop filter (ALF), can be applied to the reconstructed CU before it is placed in the reference picture store of picture buffer 120 and used for coding future video blocks. To form the output video bitstream 144, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to entropy coding 138 to be further compressed and packed to form the bitstream.

[0036] Figure 1 A block diagram of a general block-based hybrid video coding system is given. Input video signals are processed block-by-block, referred to as coding units (CUs). In VTM-1.0, a CU can be up to 128x128 pixels. However, unlike HEVC, which partitions a block based on quad-tree only, in VVC, a coding tree unit (CTU) is partitioned into CUs to adapt to the varying local characteristics based on quad-tree / binary-tree / triple-tree. In addition, the concept of multiple partition unit types in HEVC is removed, i.e., there is no longer a distinction between CU, prediction unit (PU), and transform unit (TU) in VVC; instead, each CU always serves as the basic unit for both prediction and transform without further partitioning. In the multi-type tree structure, a CTU is first partitioned by a quad-tree structure. Then, each quad-tree leaf node can be further partitioned by binary and triple tree structures.

[0037] As shown in Figure 3A , 3B , 3C, 3D, and 3E, there are five types of splitting: quad splitting, horizontal binary splitting, vertical binary splitting, horizontal triple splitting, and vertical triple splitting.

[0038] Figure 3AA diagram illustrating block horizontal ternary partitioning in a multi-type tree structure according to this disclosure.

[0039] Figure 3B A diagram illustrating block vertical binary partitioning in a multi-type tree structure according to this disclosure.

[0040] Figure 3C A diagram illustrating block horizontal binary partitioning in a multi-type tree structure according to this disclosure.

[0041] Figure 3D A diagram illustrating block vertical ternary partitioning in a multi-type tree structure according to this disclosure.

[0042] Figure 3E A diagram illustrating block horizontal ternary partitioning in a multi-type tree structure according to this disclosure.

[0043] In Figure 1 spatial prediction (or "intra prediction") uses pixels from already coded neighboring blocks in the same video picture / slice (which are referred to as reference samples) to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in video signals. Temporal prediction (also referred to as "inter prediction" or "motion-compensated prediction") uses reconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index is additionally signaled that is used to identify which reference picture from the reference picture store the temporal prediction signal comes from. After spatial and / or temporal prediction, a mode decision block in the encoder selects the best prediction mode, e.g., based on rate-distortion optimization methods. The prediction block is then subtracted from the current video block and the prediction residual is de-correlated using a transform and quantization. The quantized residual coefficients are de-quantized and inverse transformed to form the reconstructed residual, which is then added back to the prediction block to form the reconstructed signal of the CU. In addition, in-loop filters such as a deblocking filter, sample adaptive offset (SAO), and adaptive loop filter (ALF) can be applied to the reconstructed CU before it is put into the reference picture store and used for coding future video blocks. To form the output video bitstream, the coding mode (inter or intra), the prediction mode information, the motion information, and the quantized residual coefficients are all sent to an entropy encoding unit to be further compressed and packed to form the bitstream.

[0044] Figure 2 A general block diagram of a video decoder for VVC is shown. Specifically,Figure 2 A typical decoder 200 block diagram is shown. The decoder 200 has a bitstream 210, entropy decoding 212, inverse quantization 214, inverse transform 216, adder 218, intra / inter mode selection 220, intra prediction 222, memory 230, in-loop filter 228, motion compensation 224, picture buffer 226, prediction related information 234, and video output 232.

[0045] The decoder 200 is similar to the reconstruction related part in the encoder 100 belonging to the HEVC standard. Figure 1 In the decoder 200, the input video bitstream 210 is first decoded by the entropy decoding 212 to derive quantized coefficient levels and prediction related information. The quantized coefficient levels are then processed by the inverse quantization 214 and inverse transform 216 to obtain reconstructed prediction residuals. The block prediction mechanism implemented in the intra / inter mode selector 220 is configured to perform either the intra prediction 222 or the motion compensation 224 based on the decoded prediction information. The set of unfiltered reconstructed pixels is obtained by adding the reconstructed prediction residuals from the inverse transform 216 and the prediction output generated by the block prediction mechanism using the adder 218.

[0046] Before the reconstructed pixels are stored in the picture buffer 226 serving as the reference picture store, the reconstructed pixels can further go through the in-loop filter 228. The reconstructed video in the picture buffer 226 can be sent out to drive a display device, as well as for predicting future video blocks. In the case that the in-loop filter 228 is turned on, filtering operations are performed on these reconstructed pixels to derive the final reconstructed video output 232.

[0047] Figure 2 A general block diagram of a block-based video decoder is given. The video bitstream is first entropy decoded at an entropy decoding unit. The coding mode and prediction information are sent to a spatial prediction unit (if intra coded) or a temporal prediction unit (if inter coded) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit and an inverse transform unit to reconstruct a residual block. The prediction block and the residual block are then added together. The reconstructed block can further go through in-loop filtering before it is stored in a reference picture store. The reconstructed video in the reference picture store is then sent out to drive a display device, as well as for predicting future video blocks.

[0048] Bi-directional optical flow

[0049] Conventional bidirectional prediction in video encoding and decoding is a simple combination of two temporal prediction blocks obtained from a reconstructed reference image. However, due to limitations in block-based motion compensation, there can be residual small motions observable between samples in the two prediction blocks, thus reducing the efficiency of motion-compensated prediction. To address this inefficiency, for example, bidirectional optical flow (BDOF) is applied in VVC to reduce the impact of such motion on every sample within a block.

[0050] Figure 4 A diagram illustrating the BDOF model according to this disclosure is shown.

[0051] Specifically, such as Figure 4 As shown, when using bidirectional prediction, BDOF is a sample-wise motion refinement performed on top of block-based motion-compensated prediction. After applying BDOF within a 6×6 window Ω around the sub-block, the motion refinement (v) for each 4×4 sub-block is calculated by minimizing the difference between the L0 and L1 predicted samples. x v y ). Specifically, (v x v y The value of ) is derived as:

[0052]

[0053] in, It is a function that rounds down; clip3(min, max, x) is a function that clips a given value x to the range [min, max]; the symbol >> indicates a bit-by-bit right shift operation; the symbol << indicates a bit-by-bit left shift operation; th BDOF It is the motion refinement threshold to prevent propagation errors caused by irregular local motion, and it is equal to 2. 13-BD Where BD is the bit depth of the input video. In (1),

[0054] The values ​​of S1, S2, S3, S5, and S6 are calculated as follows:

[0055]

[0056] in

[0057]

[0058] Among them, I (k) (i, j) are the sample values ​​at coordinates (i, j) of the predicted signal in list k, where k = 0, 1, ...

[0059] Among them, I is generated with medium to high precision (i.e., 16 bits).(k) (i, j); and are the horizontal and vertical gradients of the sample obtained by directly calculating the difference between two neighboring samples of the sample, i.e.,

[0060]

[0061] Based on the motion refinement derived in (1), the final bi-predicted samples of a CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the optical flow model as follows:

[0062]

[0063] where shift and o offset are the right shift value and the offset value applied to combine the L0 and L1 prediction signals for bi-prediction, shift and o offset equal to 15-BD and 1 « (14-BD) + 2 · (1 « 13), respectively. Table 1 shows the specific bit-depths of the intermediate parameters involved in the BDOF process. As shown in the table, the internal bit-depth of the whole BDOF process does not exceed 32 bits. In addition, the multiplication with the worst possible input occurs at the product v x S 2，m in (1), where the inputs are 15 bits and 4 bits. Therefore, a 15-bit multiplier is sufficient for BDOF.

[0064] Table 1 Bit-depths of intermediate parameters of BDOF in VVC

[0065]

[0066]

[0067]

[0068] The terminology used in the present disclosure is for the purpose of describing example examples only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0069] It should be understood that, although the terms “first,” “second,” “third,” etc. can be used herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one category of information from another. For example, without departing from the scope of the present disclosure, first information can include what is referred to as second information; and similarly, second information can also be referred to as first information. As used herein, the term “if’ can be understood to mean “when” or “when a” or “in response to” depending on the context.

[0070] References throughout this specification to “one example,” “an example,” “exemplary example,” or similar terminology mean that a particular feature, structure, or characteristic described in connection with the example is included in at least one example of the present disclosure. Thus, the appearance of the phrases in

[0071] Current BDOF and PROF design

[0072] Although BDOF can improve the efficiency of bi-directional predictive prediction, its design can still be further improved. Specifically, the following inefficiencies in the existing BDOF design in VVC for controlling the bit-depth of the intermediate parameters are identified in the present disclosure.

[0073] 1) As shown in Table 1, the parameter 0(i,j) (i.e., the difference between the L0 prediction sample and the L1 prediction sample) and the parameter x (i,j) and the parameter y (i,j) (i.e., the sum of the horizontal L0 gradient value and the horizontal L1 gradient value / the sum of the vertical L0 gradient value and the vertical L1 gradient value). Although such an approach can facilitate the overall control of the internal bit-depth of BDOF, it is suboptimal in terms of the accuracy of the derived motion refinement. This is because, as shown in (4), the gradient value is calculated as the difference between neighboring prediction samples; due to the high-pass nature of this process, the derived gradient is less reliable in the presence of noise (e.g., noise captured in the original video and coding noise generated during the coding process). This means that it can not always be beneficial to represent the gradient value at a high bit-depth.

[0074] 2) As shown in Table 1, the maximum bit-depth usage of the entire BDOF process occurs with the calculation of the vertical motion refinement v y x S​2，m )<<12+v x S 2，s The maximum bit depth of the current design is 31 bits (30 bits). In practical hardware implementations, encoding and decoding processes with a maximum internal bit depth greater than 16 bits are typically implemented using a 32-bit implementation. Therefore, the existing design does not fully utilize the effective dynamic range of the 32-bit implementation. This can lead to unnecessary precision loss in motion refinement derived from BDOF.

[0075] Improved bit depth control method

[0076] In this disclosure, an improved bit depth control method is proposed to address two problems of existing bit depth control methods, as noted in the "Current BDOF and PROF Designs" section for existing BDOF designs. First, to overcome the negative impact of gradient estimation errors, the gradient value is calculated in (4). and The proposed method introduces an additional right shift n. grad That is, in order to reduce the internal bit depth of the gradient values. Specifically, the horizontal and vertical gradients at each sample location are calculated as follows:

[0077]

[0078] Furthermore, in order to control the entire BDOF process and operate the entire BDOF process with an appropriate internal bit depth, the additional bits are shifted by n. adj Introduced into variable ψ x (i, j), ψ y The calculation of (i, j) and θ(i, j) is as follows:

[0079]

[0080] As will be seen in Table 2, due to the modification of the number of right shift bits applied in (6) and (7), compared with the existing BDOF design in Table 1 (where the three parameters ψ are represented in the same dynamic range (i.e., 21 bits)). x (i, j), ψ y Compared to θ(i,j), the parameter ψ x (i, j), parameter ψ yThe dynamic range of (i, j) and the parameter 0(i, j) will be different. This change can increase the bit-depth of the internal parameters S1, S2, S3, S5, and S6, which can potentially increase the maximum bit-depth of the internal BDOF process to more than 32 bits. Therefore, to ensure a 32-bit implementation, two additional clipping operations are introduced when calculating the values of S2 and S6. Specifically, in the proposed method, the values of the two parameters are calculated as

[0081]

[0082] where B2 and B6 are parameters that control the output dynamic range of S2 and S6, respectively. It should be noted that, unlike the gradient calculation, the clipping operation in (8) is only applied once to calculate the motion refinement for each 4x4 sub-block within one BDOF CU (i.e., the clipping operation in (8) is invoked on a 4x4 unit basis). Therefore, the corresponding complexity increase due to the clipping operations introduced in the proposed method is completely negligible.

[0083] In practice, different values of n grad , n adj , B2, and B6 can be applied to achieve different trade-offs between the intermediate bit-depth and the internal BDOF derivation accuracy. As one embodiment of the present disclosure, it is proposed to set n grad and n adj to 2, set B2 to 25, and set B6 to 27. Table 2 shows the corresponding bit-depths of each intermediate parameter when the proposed bit-depth control method is applied to BDOF. In Table 2, the changes applied in the proposed bit-depth control method are highlighted in grey compared to the existing BDOF design in VVC. From Table 2, it can be seen that with the proposed bit-depth control method, the internal bit-depth of the entire BDOF process does not exceed 32 bits. Moreover, with the proposed design, the maximum bit-depth is exactly 32 bits, which can fully utilize the available dynamic range of a 32-bit hardware implementation. On the other hand, as shown in the table, the multiplication at v x S 2，m occurs with the multiplication of the worst possible input S 2，m , which is 14 bits, and the input v x , which is 6 bits. Therefore, as with the existing BDOF design, a 16-bit multiplier is also large enough when the proposed method is applied.

[0084] Table 2 Bit-depths of intermediate parameters of the proposed method

[0085]

[0086]

[0087]

[0088] In the above method, clipping operations as in equation (8) are added to avoid overflow of intermediate parameters when deriving v x and v y . However, such clipping is only needed when the related parameters are accumulated in a large local window. When a small window is applied, it is possible that no overflow occurs. Therefore, in another embodiment of the disclosure, the following bit-depth control method is proposed for the BDOF method without clipping, as described below.

[0089] 1) First, the gradient values in (4) at each sample position and are calculated as:

[0090]

[0091]

[0092] 2) Then, the related parameters ψ x (i, j), ψ y (i, j) and θ(i, j) for the BDOF process are calculated as:

[0093]

[0094] 3) The values of S1, S2, S3, S5 and S6 are calculated as:

[0095]

[0096] 4) The motion refinements (v s , v y ) of each 4x4 sub-block are derived as:

[0097]

[0098] 5) The final bi-prediction samples of a CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the optical flow model, as shown by the following equation:

[0099] pred BDOF (x, y) = (I (0) (x, y) + I (1) (x, y) + b + o offset ) » shift (13)

[0100]

[0101] 1) The above BDOF bit-depth control method is built on the assumption that the internal bit-depth for coding video cannot exceed 12 bits, so that the precision of the output signal from the motion compensation (MC) is 14 bits. In other words, when the internal bit-depth is larger than 12 bits, the BDOF bit-depth control method as specified in (9) to (13) cannot guarantee that all bit-depths of the internal BDOF operation are within 32 bits. To solve this overflow inefficiency for high internal bit-depth, the BDOF bit-depth control method is improved below by introducing an additional bit-wise right shift that depends on the internal bit-depth applied after the MC stage. With this method, when the internal bit-depth is larger than 12 bits, the MC output signal is always shifted to 14 bits, so that the existing BDOF bit-depth control method designed for internal bit-depths of 8 to 12 bits can be reused for the BDOF process of high bit-depth video. Specifically, assuming that the bit-depth is the internal bit-depth, the proposed method is described by the following steps: First, in (4), the gradient value at each sample position and is calculated as:

[0102]

[0103] 2) Then, the related parameters ψ x (i, j), ψ y (i, j) and θ(i, j) for the BDOF process are calculated as:

[0104]

[0105] 3) The values of S1, S2, S3, S5 and S6 are calculated as:

[0106]

[0107] 4) The motion refinement for each 4x4 sub-block is derived as:

[0108]

[0109] where th BDOF is the motion refinement threshold, which is calculated as 1 « max(5, bit-depth - 7) based on the internal bit-depth.

[0110] Figure 5 A method 500 of decoding a video signal according to the present disclosure is shown. The method can for example be applied in a decoder.

[0111] At step 510, the decoder can obtain a first reference picture I (0) and a second reference picture I (1). In display order, the first reference picture I (0) may precede the current picture and the second reference picture I (1) may follow the current picture.

[0112] At step 512, the decoder can obtain first prediction samples I (0) (i,j) of the video block from the reference block in the first reference picture I (0) (i,j). The numbers i and j can represent the coordinates of one sample within the current picture.

[0113] At step 514, the decoder can obtain second prediction samples I (1) (i,j) of the video block from the reference block in the second reference picture I (1) (i,j).

[0114] At step 516, the decoder can control the internal bit-depth of BDOF by applying a right shift to the internal BDOF parameters, wherein the BDOF is independent of the input video bit-depth, and wherein the internal BDOF parameters include horizontal gradient values and vertical gradient values derived based on the first prediction samples I (0) (i,j), horizontal gradient values and vertical gradient values derived based on the second prediction samples I (1) (i,j), and sample difference values between the first prediction samples I (0) (i,j) and the second prediction samples I (1) (i,j).

[0115] At step 518, the decoder can obtain final bi-prediction samples of the video block based on the BDOF applied to the video block from the first prediction samples I (0) (i,j) and the second prediction samples I (1) (i,j).

[0116] In the above method, although the design can ensure that the maximum intermediate bit-depth of all internal parameters for BDOF derivation does not exceed 32 bits, it can still result in different internal bit-depths for BDOF refinement derivation when the bit-depths of the input videos are different. In the following, another BDOF bit-depth control method is proposed, wherein the internal bit-depths of BDOF derivation are independent of the input video bit-depths.

[0117] 1) First, the gradient values in (4) at each sample position and are calculated as:

[0118]

[0119] 2) Then, the relevant parameters ψ x (i,j), ψy (i,j) and θ(i,j) are calculated as:

[0120]

[0121] 3) The values of S1, S2, S3, S5 and S6 are calculated as:

[0122]

[0123]

[0124]

[0125] 4) The motion refinement (v s , v y ) for each 4x4 sub-block is derived as:

[0126]

[0127] where th BDOF is the motion refinement threshold, which is a constant equal to 32.

[0128] 5) The final bi-prediction samples of the CU are calculated by interpolating the L0 prediction samples / L1 prediction samples along the motion trajectory based on the optical flow model, as shown by the following equation:

[0129]

[0130] Figure 6 A method 600 of decoding a video signal according to the present disclosure is shown. The method can be applied, for example, to a decoder.

[0131] At step 610, the decoder can obtain a horizontal gradient difference value. The horizontal gradient difference value can be a difference between a first horizontal gradient value and a second horizontal gradient value.

[0132] At step 612, the decoder can obtain a vertical gradient difference value. The vertical gradient difference value can be a difference between a first vertical gradient value and a second vertical gradient value.

[0133] At step 614, the decoder can left shift the horizontal gradient difference value by a third shift value.

[0134] At step 616, the decoder can left shift the vertical gradient difference value by the third shift value.

[0135] At step 618, the decoder can calculate the sample refinement value based on a sum of a product of the horizontal motion refinement value and the horizontal gradient difference value and a product of the vertical motion refinement value and the vertical gradient difference value. For example, the sample refinement value can be calculated based on the following equation (27). In equation (27), the sample refinement value can be b, where the sample refinement value is used to calculate pred BDOF (x, y).

[0136] At step 620, the decoder can obtain the final bi-predicted sample of the video block based on a sum of the first predicted sample I (0) (i, j), the second predicted sample I (1) (i, j), the sample refinement value, and the offset value.

[0137] At step 622, the decoder can right shift the final bi-predicted sample by a fourth shift value.

[0138] In the current VVC design, the precision of the output prediction signal is not constant when the internal bit-depth is larger than 12 bits. Such design can not be very friendly for the following BDOF derivation. In one embodiment of the present disclosure, a new bit-shifting method is proposed for the motion compensation prediction for high internal bit-depth (i.e., > 12 bits) to align the precision of the output prediction signal to a constant (e.g., 20 bits). For example, the proposed method can include at least the following steps:

[0139] 1) Perform horizontal interpolation using the reference samples from the temporal reference picture to obtain the horizontally fractional interpolated prediction sample, then, when the internal bit-depth is not larger than 12 bits, apply a right shift of (bit-depth - 8) to the interpolated prediction sample. In addition, if the internal bit-depth is larger than 12 bits, apply a left shift of (bit-depth - 12) to the interpolated prediction sample.

[0140] 2) Perform vertical interpolation using the interpolated prediction sample from the first step, apply a right shift of 6 bits.

[0141] Given the fixed precision of the interpolated prediction sample, the following unified BDOF bit-depth control method can be applied for high internal bit-depth beyond 12 bits:

[0142] 1) First, the gradient values in (4) at each sample position and are calculated as:

[0143]

[0144] 2) Then, the related parameters ψ x (i, j), ψy (i,j) and θ(i,j) are computed as:

[0145]

[0146]

[0147] θ(i,j) = (I (1) (i,j) > 8) - (I (0) (i,j) > 8)

[0148] 3) The values of S1, S2, S3, S5 and S6 are computed as:

[0149]

[0150] 4) The motion refinement (v s , v y ) for each 4x4 sub-block is derived as:

[0151]

[0152] where th BDOF is the motion refinement threshold, which is a constant equal to 32.

[0153] 5) The final bi-prediction samples for a CU are computed by interpolating the L0 prediction samples / L1 prediction samples along the motion trajectory based on the optical flow model, as shown by the following equation:

[0154]

[0155] The above-described methods can be implemented using an apparatus including one or more circuits such as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components. The apparatus can use circuits in combination with other hardware or software components to perform the above-described methods. Each of the modules, sub-modules, units or sub-units disclosed above can be implemented at least partly using the one or more circuits.

[0156] Figure 7 A computing environment 710 is shown coupled with a user interface 760. The computing environment 710 can be part of a data processing server. The computing environment 710 includes a processor 720, a memory 740, and an I / O interface 750.

[0157] The processor 720 generally controls the overall operation of the computing environment 710, such as operations associated with displaying, data acquisition, data communication, and image processing. The processor 720 can include one or more processors for executing instructions to perform all or some of the steps in the above methods. In addition, the processor 720 can include one or more modules that facilitate interaction with other components of the computing environment 710. The processor can be a central processing unit (CPU), a microprocessor, a microcomputer, a GPU, etc.

[0158] The memory 740 is configured to store various types of data to support the operation of the computing environment 710. The memory 740 can include predetermined software 742. Examples of such data include instructions for any application or method operating on the computing environment 710, video data sets, image data, etc. The memory 740 can be implemented by using any type of volatile memory device or non-volatile memory device, or a combination of volatile memory device and non-volatile memory device, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0159] The I / O interface 750 provides an interface between the processor 720 and peripheral interface modules such as a keyboard, a click wheel, a button, etc. The button can include, but is not limited to, a home button, a start scanning button, and a stop scanning button. The I / O interface 750 can be coupled with an encoder and a decoder.

[0160] In some embodiments, a non-transitory computer-readable storage medium including a plurality of programs, such as included in the memory 740, is also provided, which can be executed by the processor 720 in the computing environment 710 for performing the above methods. For example, the non-transitory computer-readable storage medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0161] A plurality of programs is stored in a non-transitory computer-readable storage medium, which is executed by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, causes the computing device to perform the above method for motion prediction.

[0162] In some embodiments, computing environment 710 can implement the above-described methods with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), controllers, microcontrollers, microprocessors, or other electronic components, performing some or all of the processing described above.

[0163] The description of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications, variations, and alternatives of the present disclosure will be apparent to those having ordinary skill in the art based on the teachings and associated drawings presented above.

[0164] The examples chosen and described are for the purpose of explanation and to enable others skilled in the art to make and use the present disclosure, and best utilize the basic principles and various embodiments with various modifications as are suited to the particular use contemplated. Therefore, it is to be understood that the scope of the present disclosure is not to be limited to the specific examples disclosed, and that modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A bit-depth control method for bi-directional optical flow (BDOF) of a video signal, comprising: obtaining, at a decoder, a first reference picture associated with a video block and a second reference picture wherein, in display order, the first reference picture precedes a current picture and the second reference picture follows the current picture; at the decoder from a first reference picture obtaining first prediction samples of the video block from a reference block in the first reference picture where i and j represent coordinates of a sample within the current picture. obtaining, at the decoder, a second prediction sample of the video block from a reference block in the second reference picture ;​ controlling an internal bit depth of the BDOF at the decoder by applying a right shift to an internal BDOF parameter, wherein the BDOF is independent of an input video bit depth, wherein a shift value of the right shift is determined based on the internal bit depth, and wherein the internal BDOF parameter comprises a sample difference between the first prediction sample a horizontal gradient value and a vertical gradient value derived based on the first prediction sample a horizontal gradient value and a vertical gradient value derived based on the first prediction sample and the second prediction sample ; and are applied to the video block to obtain final bi-predicted samples of the video block based on the BDOF according to the first prediction samples and the second prediction samples are applied to the video block to obtain final bi-predicted samples of the video block.

2. The method of claim 1, wherein, controlling an internal bit-depth of the BDOF by applying a right shift to internal BDOF parameters comprises: Based on the first prediction sample at the decoder Compared with the first predicted sample The difference between them is used to obtain the first prediction sample. The first horizontal gradient value; obtaining, at the decoder, a second horizontal gradient value of the second prediction sample based on a difference between the second prediction sample and a first prediction sample and the first prediction sample obtaining, at the decoder, a second horizontal gradient value of the second prediction sample based on a difference between the second prediction sample and a first prediction sample and the first prediction sample at the decoder based on a difference between the first prediction sample and a second prediction sample to obtain a first vertical gradient value of the first prediction sample ​ obtaining, at the decoder, a second vertical gradient value of the second prediction sample based on a difference between the second prediction sample and a third prediction sample and the third prediction sample obtained based on a difference between the second prediction sample and a fourth prediction sample obtained based on a difference between the second prediction sample and a fifth prediction sample right shifting, at the decoder, the first horizontal gradient value and the second horizontal gradient value by a first shift value; and right shifting, at the decoder, the first vertical gradient value and the second vertical gradient value by the first shift value.

3. The method of claim 2, wherein, the first shift value is equal to the internal bit-depth minus 6.

4. The method of claim 1, further comprising: obtaining, at the decoder, a first correlation value, wherein the first correlation value is based on a sum of horizontal gradient values of the first prediction samples and based on a sum of horizontal gradient values of the second prediction samples ​ obtaining a second correlation value at the decoder, wherein the second correlation value is a sum based on vertical gradient values of the first prediction samples and based on vertical gradient values of the second prediction samples ​ modifying, at the decoder, the first correlation value by right shifting the first correlation value by 1; and modifying, at the decoder, the second correlation value by right shifting the second correlation value by 1.

5. The method of claim 4, further comprising: at the decoder by using a second shift value to obtain a first modified prediction sample from the first prediction sample right shift obtaining a second modified prediction sample at the decoder by using the second shift value to right shift the second prediction sample obtaining a second modified prediction sample at the decoder by using the second shift value to right shift the second prediction sample and obtaining, at the decoder, a third correlation value, wherein the third correlation value is a difference between the first modified prediction sample and the second modified prediction sample.

6. The method of claim 5, wherein, the second shift value is equal to the internal bit-depth minus 8.

7. The method of claim 5, further comprising: obtaining, at the decoder, a plurality of internal sum values based on the first, second, and third correlation values within each 4x4 sub-block of the video block.

8. The method of claim 7, further comprising: obtaining, at the decoder, a horizontal motion refinement value based on at least one of the plurality of internal sum values, wherein the motion refinement values include the horizontal motion refinement value; obtaining, at the decoder, a vertical motion refinement value based on at least one of the plurality of internal sum values and the horizontal motion refinement value, wherein the motion refinement values include the vertical motion refinement value; and clipping, at the decoder, the horizontal motion refinement value and the vertical motion refinement value based on a motion refinement threshold.

9. The method of claim 8, further comprising: obtaining, at the decoder, a horizontal gradient difference value, wherein the horizontal gradient difference value is a difference between a first horizontal gradient value and a second horizontal gradient value; obtaining, at the decoder, a vertical gradient difference value, wherein the vertical gradient difference value is a difference between a first vertical gradient value and a second vertical gradient value; left shifting, at the decoder, the horizontal gradient difference value by a third shift value; left shifting, at the decoder, the vertical gradient difference value by the third shift value; calculating, at the decoder, a sample refinement value based on a sum of a product of the horizontal motion refinement value and the horizontal gradient difference value and a product of the vertical motion refinement value and the vertical gradient difference value; at the decoder based on the first prediction sample , the second prediction sample , the sum of the sample refinement value and an offset value to obtain the final bi-prediction sample of the video block; and right shifting, at the decoder, the final bi-directional prediction sample by a fourth shift value.

10. The method of claim 9, wherein, the third shift value is equal to the internal bit-depth minus 12.

11. A bit-depth control method for bi-directional optical flow (BDOF) of a video signal, comprising: obtaining, at a decoder, a first reference picture associated with a video block and a second reference picture wherein, in display order, the first reference picture precedes a current picture and the second reference picture follows the current picture; at the decoder from a reference block in the first reference picture obtaining first prediction samples of the video block where i and j represent coordinates of a sample within the current picture; obtaining, at the decoder, a second prediction sample of the video block from a reference block in the second reference picture ;​ at the decoder and when an internal bit depth is greater than 12 bits, controlling the internal bit depth of the BDOF by applying a right shift to internal BDOF parameters to align the precision of an output prediction signal to a constant, wherein the BDOF is independent of an input video bit depth, and wherein the internal BDOF parameters include a sample difference between the first prediction sample derived horizontal gradient value and a vertical gradient value, a horizontal gradient value and a vertical gradient value derived based on the second prediction sample derived horizontal gradient value and a vertical gradient value, and the first prediction sample and the second prediction sample ​ At the decoder, based on the BDOF and according to the first prediction sample and the second predicted sample The data is applied to the video block to obtain the final bidirectional prediction samples of the video block; and obtaining, at the decoder, the output prediction signal based on the final bi-directional prediction sample.

12. The method of claim 11, wherein, controlling the internal bit depth of the BDOF by aligning the precision of the output prediction signal to a constant by applying a right shift to the internal BDOF parameters when the internal bit depth is greater than 12 bits comprises: Based on the first prediction sample at the decoder Compared with the first predicted sample The difference between them is used to obtain the first prediction sample. The first horizontal gradient value; obtaining, at the decoder, a second horizontal gradient value of the second prediction sample based on a difference between the second prediction sample and a first prediction sample and the first prediction sample ​​ at the decoder based on a difference between the first prediction sample and a second prediction sample to obtain a first vertical gradient value of the first prediction sample ​ obtaining, at the decoder, a second vertical gradient value of the second prediction sample based on a difference between the second prediction sample and a second reference sample of the second reference line and the second prediction sample obtaining, at the decoder, a second vertical gradient value of the second prediction sample based on a difference between the second prediction sample and a second reference sample of the second reference line and the second prediction sample right shifting the first horizontal gradient value and the second horizontal gradient value by 10 at the decoder; and right shifting the first vertical gradient value and the second vertical gradient value by 10 at the decoder.

13. The method of claim 11, further comprising: obtaining, at the decoder, a first correlation value, wherein the first correlation value is based on a sum of horizontal gradient values of the first prediction samples and based on a sum of horizontal gradient values of the second prediction samples ​ obtaining a second correlation value at the decoder, wherein the second correlation value is a sum based on vertical gradient values of the first prediction samples and based on vertical gradient values of the second prediction samples ​ modifying the first correlation value by right shifting the first correlation value by 1 at the decoder; and modifying the second correlation value by right shifting the second correlation value by 1 at the decoder.

14. The method of claim 13, further comprising: at the decoder by obtaining a first modified prediction sample by 8 adding the first prediction sample to the first residual sample; at the decoder by shifting the second prediction samples right by 8 to obtain second modified prediction samples; and obtaining a third correlation value at the decoder, wherein the third correlation value is a difference between the first modified prediction sample and the second modified prediction sample.

15. The method of claim 14, further comprising: obtaining a plurality of internal sum values at the decoder based on the first correlation value, the second correlation value, and the third correlation value within each 4x4 subblock of the video block.

16. The method of claim 15, further comprising: obtaining a horizontal motion refinement value at the decoder based on at least one of the plurality of internal sum values, wherein the motion refinement values include the horizontal motion refinement value; obtaining a vertical motion refinement value at the decoder based on at least one of the plurality of internal sum values and the horizontal motion refinement value, wherein the motion refinement values include the vertical motion refinement value; and clipping the horizontal motion refinement value and the vertical motion refinement value at the decoder based on a motion refinement threshold.

17. The method of claim 16, further comprising: obtaining a horizontal gradient difference value at the decoder, wherein the horizontal gradient difference value is a difference between a first horizontal gradient value and a second horizontal gradient value; obtaining a vertical gradient difference value at the decoder, wherein the vertical gradient difference value is a difference between a first vertical gradient value and a second vertical gradient value; left shifting the horizontal gradient difference value by 4 at the decoder; left shifting the vertical gradient difference value by 4 at the decoder; calculating a sample refinement value based on a sum of a product of the horizontal motion refinement value and the horizontal gradient difference value and a product of the vertical motion refinement value and the vertical gradient difference value; at the decoder based on the first prediction sample , the second prediction sample , the sum of the sample refinement value and an offset value to obtain the final bi-prediction sample of the video block; and right shifting the final bi-predicted sample by a shift value at the decoder.

18. A computing device, comprising: one or more processors; a non-transitory computer-readable storage medium storing instructions executable by the one or more processors, wherein the one or more processors are configured to: obtaining, at a decoder, a first reference picture associated with a video block and a second reference picture wherein, in display order, the first reference picture precedes a current picture and the second reference picture follows the current picture; at the decoder from a reference block in the first reference picture obtaining first prediction samples of the video block where i and j represent coordinates of a sample within the current picture; obtaining, at the decoder, a second prediction sample of the video block from a reference block in the second reference picture ;​ controlling an internal bit-depth of bi-directional optical flow (BDOF) at the decoder by applying a right shift to internal BDOF parameters, wherein the BDOF is independent of an input video bit-depth, wherein a shift value of the right shift is determined based on the internal bit-depth, and wherein the internal BDOF parameters include a sample difference between the first prediction sample derived horizontal gradient value and a vertical gradient value, a second prediction sample derived horizontal gradient value and a vertical gradient value, and the first prediction sample and the second prediction sample ​ At the decoder, based on the BDOF and according to the first prediction sample and the second predicted sample It is applied to the video block to obtain the final bidirectional prediction sample of the video block.

19. The computing device of claim 18, wherein, the one or more processors configured to control the internal bit depth of the BDOF by aligning the precision of the output prediction signal to a constant by applying a right shift to the internal BDOF parameters further configured to: the one or more processors configured to control the internal bit depth of the BDOF by aligning the precision of the output prediction signal to a constant by applying a right shift to the internal BDOF parameters further configured to: Based on the first prediction sample at the decoder Compared with the first predicted sample The difference between them is used to obtain the first prediction sample. The first horizontal gradient value; obtaining, at the decoder, a second horizontal gradient value of the second prediction sample based on a difference between the second prediction sample and a first prediction sample and the first prediction sample obtained at the encoder ​ at the decoder based on a difference between the first prediction sample and a second prediction sample to obtain a first vertical gradient value of the first prediction sample ​ obtaining, at the decoder, a second vertical gradient value of the second prediction sample based on a difference between the second prediction sample and a third prediction sample and the third prediction sample obtained based on a difference between the second prediction sample and a fourth prediction sample obtained based on a difference between the second prediction sample and a fifth prediction sample right shifting, at the decoder, the first horizontal gradient value and the second horizontal gradient value by a first shift value; and right shifting, at the decoder, the first vertical gradient value and the second vertical gradient value by the first shift value.

20. The computing device of claim 19, wherein, The first shift value is equal to the internal bit depth minus 6.

21. The computing device of claim 18, wherein, The one or more processors are further configured to: obtaining, at the decoder, a first correlation value, wherein the first correlation value is based on a sum of horizontal gradient values of the first prediction samples and based on a sum of horizontal gradient values of second prediction samples ​ obtaining a second correlation value at the decoder, wherein the second correlation value is a sum based on vertical gradient values of the first prediction samples and based on vertical gradient values of the second prediction samples ​ modify, at the decoder, the first correlation value by right shifting the first correlation value by 1; and modify, at the decoder, the second correlation value by right shifting the second correlation value by 1.

22. The computing device of claim 21, wherein, The one or more processors are further configured to: at the decoder by using a second shift value to obtain a first modified prediction sample from the first prediction sample right shift obtaining a second modified prediction sample at the decoder by using the second shift value to right shift the second prediction sample obtaining a second modified prediction sample at the decoder by using the second shift value to right shift the second prediction sample and obtain, at the decoder, a third correlation value, wherein the third correlation value is a difference between the first modified prediction sample and the second modified prediction sample.

23. The computing device of claim 22, wherein, The second shift value is equal to the internal bit depth minus 8.

24. The computing device of claim 22, wherein, The one or more processors are further configured to: obtain, at the decoder, a plurality of internal sum values based on the first correlation value, the second correlation value, and the third correlation value within each 4x4 sub-block of the video block.

25. The computing device of claim 24, further comprising: obtain, at the decoder, a horizontal motion refinement value based on at least one of the plurality of internal sum values, wherein the motion refinement values include the horizontal motion refinement value; obtain, at the decoder, a vertical motion refinement value based on at least one of the plurality of internal sum values and the horizontal motion refinement value, wherein the motion refinement values include the vertical motion refinement value; and clip, at the decoder, the horizontal motion refinement value and the vertical motion refinement value based on a motion refinement threshold.

26. The computing device of claim 25, wherein, The one or more processors are further configured to: obtain, at the decoder, a horizontal gradient difference value, wherein the horizontal gradient difference value is a difference between a first horizontal gradient value and a second horizontal gradient value; obtain, at the decoder, a vertical gradient difference value, wherein the vertical gradient difference value is a difference between a first vertical gradient value and a second vertical gradient value; left shift, at the decoder, the horizontal gradient difference value by a third shift value; left shift, at the decoder, the vertical gradient difference value by the third shift value; calculate, at the decoder, a sample refinement value based on a sum of a product of the horizontal motion refinement value and the horizontal gradient difference value and a product of the vertical motion refinement value and the vertical gradient difference value; at the decoder based on the first prediction sample , the second prediction sample the sum of the sample refinement value and the offset value to obtain the final bi-prediction sample of the video block; and right shift, at the decoder, the final bi-predicted sample by a fourth shift value.

27. The computing device of claim 26, wherein, The third shift value is equal to the internal bit depth minus 12.

28. A non-transitory computer-readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, wherein, The plurality of programs, when executed by the one or more processors, cause the computing device to perform acts comprising: obtaining, at a decoder, a first reference picture associated with a video block and a second reference picture wherein, in display order, the first reference picture precedes a current picture and the second reference picture follows the current picture; at the decoder from a reference block in the first reference picture obtaining first prediction samples of the video block where i and j represent coordinates of a sample within the current picture; obtaining, at the decoder, a second prediction sample of the video block from a reference block in the second reference picture ;​ At the decoder and when an internal bit depth is greater than 12 bits, controlling the internal bit depth of bi-directional optical flow (BDOF) by applying a right shift to internal BDOF parameters to align the precision of an output prediction signal to a constant, wherein the BDOF is independent of an input video bit depth, and wherein the internal BDOF parameters include a sample difference between the first prediction sample derived horizontal gradient value and a vertical gradient value, a second prediction sample derived horizontal gradient value and a vertical gradient value, and the first prediction sample derived horizontal gradient value and a vertical gradient value, and the first prediction sample​ applying, at the decoder, the BDOF based on the first prediction samples and the second prediction samples to the video block to obtain final bi-prediction samples of the video block; and obtain, at the decoder, the output prediction signal based on the final bi-predicted sample.

29. The non-transitory computer-readable storage medium of claim 28, wherein, The plurality of programs further cause the computing device to perform acts comprising: Based on the first prediction sample at the decoder Compared with the first predicted sample The difference between them is used to obtain the first prediction sample. The first horizontal gradient value; obtaining, at the decoder, a second horizontal gradient value of the second prediction sample based on a difference between the second prediction sample and a first prediction sample and the first prediction sample obtained at the encoder ​ at the decoder based on a difference between the first prediction sample and a second prediction sample to obtain a first vertical gradient value of the first prediction sample ​ obtaining, at the decoder, a second vertical gradient value of the second prediction sample based on a difference between the second prediction sample and a second reference sample of the second reference line and the second prediction sample obtaining, at the decoder, a second vertical gradient value of the second prediction sample based on a difference between the second prediction sample and a second reference sample of the second reference line and the second prediction sample right shifting, at the decoder, the first horizontal gradient value and the second horizontal gradient value by 10; and right shifting, at the decoder, the first vertical gradient value and the second vertical gradient value by 10.

30. The non-transitory computer-readable storage medium of claim 28, wherein, The plurality of programs further cause the computing device to perform the following acts: obtaining, at the decoder, a first correlation value, wherein the first correlation value is based on a sum of horizontal gradient values of the first prediction samples and based on a sum of horizontal gradient values of the second prediction samples ​ obtaining a second correlation value at the decoder, wherein the second correlation value is a sum based on vertical gradient values of the first prediction samples and based on vertical gradient values of the second prediction samples ​ modifying, at the decoder, the first correlation value by right shifting the first correlation value by 1; and modifying, at the decoder, the second correlation value by right shifting the second correlation value by 1.

31. The non-transitory computer-readable storage medium of claim 30, wherein, The plurality of programs further cause the computing device to perform the following acts: at the decoder by obtaining a first modified prediction sample by 8 adding the first prediction sample to the first residual sample; at the decoder by shifting the second prediction samples right by 8 to obtain second modified prediction samples; and obtaining, at the decoder, a third correlation value, wherein the third correlation value is a difference between the first modified prediction sample and the second modified prediction sample.

32. The non-transitory computer-readable storage medium of claim 31, wherein, The plurality of programs further cause the computing device to perform the following acts: obtaining, at the decoder, a plurality of intra-sum values based on the first correlation value, the second correlation value, and the third correlation value within each 4x4 sub-block of the video block.

33. The non-transitory computer-readable storage medium of claim 32, further comprising: obtaining, at the decoder, a horizontal motion refinement value based on at least one of the plurality of intra-sum values, wherein the motion refinement values include the horizontal motion refinement value; obtaining, at the decoder, a vertical motion refinement value based on at least one of the plurality of intra-sum values and the horizontal motion refinement value, wherein the motion refinement values include the vertical motion refinement value; and clipping, at the decoder, the horizontal motion refinement value and the vertical motion refinement value based on a motion refinement threshold.

34. The non-transitory computer-readable storage medium of claim 33, wherein, The plurality of programs further cause the computing device to perform the following acts: obtaining, at the decoder, a horizontal gradient difference value, wherein the horizontal gradient difference value is a difference between a first horizontal gradient value and a second horizontal gradient value; obtaining, at the decoder, a vertical gradient difference value, wherein the vertical gradient difference value is a difference between a first vertical gradient value and a second vertical gradient value; left shifting, at the decoder, the horizontal gradient difference value by 4; left shifting, at the decoder, the vertical gradient difference value by 4; calculating, at the decoder, a sample refinement value based on a sum of a product of the horizontal motion refinement value and the horizontal gradient difference value and a product of the vertical motion refinement value and the vertical gradient difference value; at the decoder based on the first prediction sample , the second prediction sample , the sum of the sample refinement value and an offset value to obtain the final bi-prediction sample of the video block; and right shifting, at the decoder, the final bi-predicted sample by a shift value.

Images