Method and device for bitwidth control for bidirectional optical flow - Patents.com

By adjusting the bit width of intermediate parameters in BDOF through right shifts and clipping, the method improves the precision and efficiency of bidirectional predictive prediction in video encoding, addressing the inefficiencies of existing designs.

JP7586988B2Active Publication Date: 2024-11-19BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023156482
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-03-15
Filing Date
2023-09-21
Publication Date
2024-11-19
Estimated Expiration
2040-03-16

AI Technical Summary

Technical Problem

Existing bidirectional optical flow (BDOF) designs in video encoding, such as those used in VVC, suffer from inefficiencies due to suboptimal bit width control of intermediate parameters, leading to potential precision loss and underutilization of hardware capabilities.

Method used

A method for controlling the internal bit width of BDOF by adjusting the bit width of intermediate parameters through additional right shifts and clipping operations, ensuring the process operates within the dynamic range of 32-bit hardware implementations.

Benefits of technology

The proposed method enhances the precision and efficiency of bidirectional predictive prediction by optimizing the bit width control, fully utilizing hardware capabilities and reducing precision loss.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586988000026
    Figure 0007586988000026
  • Figure 0007586988000027
    Figure 0007586988000027
  • Figure 0007586988000028
    Figure 0007586988000028
Patent Text Reader

Abstract

To provide a bit-width control method of a bi-directional optical flow (BDOF).SOLUTION: The method includes: obtaining a first reference picture I(0) and a second reference picture I(1) that are associated with a video block; obtaining first prediction samples I(0)(i,j) of the video block from a reference block in the first reference picture I(0); obtaining second prediction samples I(1)(i,j) of the video block from a reference block in the second reference picture I(1); controlling internal bit-widths of the BDOF by deriving internal bit-widths of intermediate parameters; obtaining motion refinements for samples in the video block based on the BDOF applied to the video block based on the first prediction samples I(0)(i,j) and the second prediction samples I(1)(i,j); and obtaining bi-prediction samples of the video block based on the motion refinements.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. Provisional Patent Application No. 62 / 819,408, filed March 15, 2019, the entire contents of which are incorporated herein by reference in their entirety.

[0002] This application relates to video encoding and compression. More particularly, this disclosure relates to a method and apparatus for bidirectional optical flow (BDOF) methods for video encoding. [Background technology]

[0003] Various video coding techniques can be used to compress video data. Video coding is performed according to one or more video coding standards. For example, video coding standards include Versatile Video Coding (VVC), Joint Exploration Model Coding (JEM), High Efficiency Video Coding (H.265 / HEVC), Advanced Video Coding (H.264 / AVC), Moving Picture Experts Group (MPEG) coding, and others. Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit redundancy present in a video image or sequence. An important goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing degradation of video quality. Summary of the Invention

[0004] According to a first aspect of the present disclosure, a bitwidth control method for encoding a video signal is provided. The method includes: (0) and the second reference picture I (1) In the display order, the first reference picture I (0)can come before the current picture, and the second reference picture I (1) can come after the current picture. This method starts with the first reference picture I (0) The first predicted sample of a video block from a reference block in (0) (i,j). The variables i and j may represent the coordinates of one sample relative to the current picture. The method may further include obtaining the second reference picture I (1) The second predicted sample of video block I from a reference block in (1) (i,j). The method may further include controlling an internal bit-width of the BDOF by deriving an internal bit-width of the intermediate parameter. The intermediate parameter is a first predicted sample I (0) (i,j) and the second prediction sample I (1) (i,j) and the intermediate parameters include a horizontal gradient value, a vertical gradient value, and a sample difference between the first prediction sample I (0) (i,j) and the second prediction sample I (1) The method can further include obtaining motion compensation for samples within the video block based on a BDOF applied to the video block based on (i,j). The method can further include obtaining bi-directionally predicted samples of the video block based on the motion compensation.

[0005] According to a second aspect of the present disclosure, there is provided a bitwidth control method for encoding a video signal, the method comprising: (0) and the second reference picture I (1) In the display order, the first reference picture I (0) can come before the current picture, and the second reference picture I (1) can come after the current picture. This method starts with the first reference picture I (0) The first predicted sample of a video block from a reference block in (0)(i,j), where the variables i and j may represent the coordinates of one sample relative to the current picture. This method may include obtaining a second reference picture I (1) The second predicted sample of video block I from a reference block in (1) (i,j). The method can include controlling an internal bit-width of the BDOF by deriving an internal bit-width of the intermediate parameter. The intermediate parameter is a first predicted sample I (0) (i,j) and the second prediction sample I (1) (i,j) and the intermediate parameters include a horizontal gradient value, a vertical gradient value, and a sample difference between the first prediction sample I (0) (i,j) and the second prediction sample I (1) (i,j) to obtain motion compensation for samples within the video block based on the BDOF and a local window applied to the video block using (i,j), where the local window includes the video block, and the local window has a window size of less than 6 by 6 pixels. The method can include obtaining bi-directionally predicted samples of the video block based on the motion compensation.

[0006] According to a third aspect of the present disclosure, a computing device is provided. The computing device may include one or more processors and a non-transitory computer-readable memory storing instructions executable by the one or more processors. The one or more processors may generate a first reference picture I associated with a video block. (0) and the second reference picture I (1) In the display order, the first reference picture I (0) can come before the current picture, and the second reference picture I (1) The one or more processors may select the first reference picture I (0) The first predicted sample of a video block from a reference block in(0) (i,j), where the variables i and j may represent the coordinates of one sample relative to the current picture. The one or more processors may be configured to obtain the second reference picture I (1) The second predicted sample of video block I from a reference block in (1) The one or more processors may be configured to obtain an internal bit-width of the bidirectional optical flow (BDOF) by deriving an internal bit-width of the intermediate parameter. The intermediate parameter may be a first prediction sample I (0) (i,j) and the second prediction sample I (1) The one or more processors may further include a horizontal gradient value, a vertical gradient value, and a sample difference between the first prediction sample I(i,j). (0) (i,j) and the second prediction sample I (1) The one or more processors may be configured to obtain bi-directionally predicted samples for the video block based on the motion compensation, the one or more processors may be configured to obtain bi-directionally predicted samples for the video block based on the motion compensation.

[0007] According to a fourth aspect of the present disclosure, a computing device is provided. The computing device may include one or more processors and a non-transitory computer-readable memory storing instructions executable by the one or more processors. The one or more processors may generate a first reference picture I associated with a video block. (0) and the second reference picture I (1) In the display order, the first reference picture I (0) can come before the current picture, and the second reference picture I (1) The one or more processors may select the first reference picture I (0) The first predicted sample of a video block from a reference block in (0)(i,j), where the variables i and j may represent the coordinates of one sample relative to the current picture. The one or more processors may be configured to obtain the second reference picture I (1) The second predicted sample of video block I from a reference block in (1) The one or more processors may be configured to obtain an internal bit-width of the bidirectional optical flow (BDOF) by deriving an internal bit-width of the intermediate parameter. The intermediate parameter may be a first prediction sample I (0) (i,j) and the second prediction sample I (1) The one or more processors may include intermediate parameters comprising a horizontal gradient value, a vertical gradient value, and a sample difference between the first prediction sample I(i,j). (0) (i,j) and the second prediction sample I (1) The one or more processors may be configured to obtain bi-directional predicted samples for the video block based on the BDOF and a local window applied to the video block using (i,j). The local window may include a sub-block, and the local window may have a window size of 6 by 6 pixels. The one or more processors may be configured to obtain bi-directional predicted samples for the video block based on the motion compensation.

[0008] It is to be understood that both the foregoing general description and the following detailed description are merely examples and are not restrictive of the present disclosure.

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram of an encoder according to an example of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of a decoder according to an example of the present disclosure. [Figure 3A] FIG. 2 illustrates a diagram showing block partitions within multiple types of tree structures according to an example of the present disclosure. [Figure 3B] FIG. 2 illustrates a diagram showing block partitions within multiple types of tree structures according to an example of the present disclosure. [Figure 3C] FIG. 2 illustrates a diagram showing block partitions within multiple types of tree structures according to an example of the present disclosure. [Figure 3D] FIG. 2 illustrates a diagram showing block partitions within multiple types of tree structures according to an example of the present disclosure. [Figure 3E] FIG. 2 illustrates a diagram showing block partitions within multiple types of tree structures according to an example of the present disclosure. [Figure 4] FIG. 1 is an illustration of a bidirectional optical flow (BDOF) model according to an example of the present disclosure. [Diagram 5] 4 is a flow diagram illustrating a bitwidth control method for encoding a video signal according to an example of the present disclosure. [Figure 6] 1 is a flow diagram illustrating a BDOF bitwidth control method according to an example of the present disclosure. [Figure 7] FIG. 1 illustrates a computing environment coupled to a user interface according to an example of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following description of the embodiments do not represent all implementations consistent with the present disclosure. Instead, these implementations are merely examples of apparatuses and methods consistent with aspects related to the present disclosure described in the accompanying claims.

[0012] The terms used in this disclosure are only intended to describe certain embodiments and are not intended to limit the disclosure. When used in this disclosure and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is also understood that the term "and / or" used herein means and includes any possible combination of one or more of the associated listed items.

[0013] Terms such as "first," "second," and "third" may be used herein to describe various pieces of information, but it should be understood that the information should not be limited by these terms. These terms are used only to distinguish one category of information from another category of information. For example, a first piece of information may be referred to as a second piece of information, and similarly, a second piece of information may be referred to as a first piece of information, without departing from the scope of this disclosure. As used herein, the term "if" will be understood to mean "when" or "upon" or "in response to a judgment," depending on the context.

[0014] The first version of the HEVC standard was finalized in October 2013 and offers approximately 50% bitrate savings or equivalent perceptual quality when compared to the previous generation video coding standard H.264 / MPEG AVC. Although the HEVC standard offers significant coding improvements over its predecessor, it has been proven that better coding efficiency than HEVC can be achieved with additional coding tools. Based on that, both VCEG and MPEG have started the search work for new coding techniques for future video coding standardization. In October 2015, a Joint Video Exploration Team (JVET) was formed by ITU-T VECG and ISO / IEC MPEG to start a significant study of advanced techniques that may enable substantial enhancement of coding efficiency. A reference software called Joint Exploration Model (JEM) was claimed by JVET by integrating several additional coding tools in addition to the HEVC Test Model (HM).

[0015] In October 2017, a Joint Call for Proposals (CfP) on video compression with capabilities beyond HEVC was published by ITU-T and ISO / IEC [9]. In April 2018, 23 CfP responses were accepted and evaluated at the 10th JVET meeting, demonstrating approximately 40% compression efficiency improvement over HEVC. Based on such evaluation results, JVET embarked on a new project to develop a new generation video coding standard called Versatile Video Coding (VVC)

[10] . In the same month, one reference software code base called the VVC Test Model (VTM)

[11] was established to demonstrate a reference implementation of the VVC standard.

[0016] Similar to HEVC, VVC is built on a block-based hybrid video coding framework. FIG. 1 shows a block diagram of a generic block-based hybrid video coding system. An input video signal is processed block by block (called a coding unit (CU)). Specifically, FIG. 1 shows an exemplary encoder 100 according to the present disclosure. The encoder 100 has a video input 110, motion compensation 112, motion estimation 114, intra / inter mode decision 116, block predictor 140, adder 128, transform 130, quantization 132, prediction relationship information 142, intra prediction 118, picture buffer 120, inverse quantization 134, inverse transform 136, adder 126, memory 124, in-loop filter 122, entropy coding 138, and bitstream 144.

[0017] In VTM-1.0, a CU can be up to 128x128 pixels. However, unlike HEVC, which splits blocks based only on quadtrees, in VVC, one coding tree unit (CTU) is split into multiple CUs to adapt to varying local features based on quadtrees / binary trees / tertiles. In addition, the concept of multiple partition unit types in HEVC is removed, i.e., the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC, and instead each CU is always used as a base unit for both prediction and transformation without further splitting. In the multiple types of tree structure, first one CTU is split by a quadtree structure. Then the leaf nodes of each quadtree may be further split by binary and tertile structures.

[0018] As shown in Figures 3A, 3B, 3C, 3D, and 3E (described below), there are five split types: 4-split, 2-split horizontal, 2-split vertical, 3-split horizontal, and 3-split vertical.

[0019] FIG. 3A shows a diagram illustrating the quadrant of blocks in multiple types of tree structures according to the present disclosure.

[0020] FIG. 3B shows a diagram illustrating vertical bisection of blocks in multiple types of tree structures according to the present disclosure.

[0021] FIG. 3C shows a diagram illustrating horizontal bisection of blocks in multiple types of tree structures according to the present disclosure.

[0022] FIG. 3D shows a diagram illustrating a vertical third division of blocks in multiple types of tree structures according to the present disclosure.

[0023] FIG. 3E shows a diagram illustrating a horizontal third division of blocks in multiple types of tree structures according to the present disclosure.

[0024] In FIG. 1, spatial prediction and / or temporal prediction may be performed. Spatial prediction (or "intra prediction") predicts a current video block using pixels from samples of already coded neighboring blocks (called reference samples) in the same video picture / slice. Spatial prediction reduces spatial redundancy inherent in video signals. Temporal prediction (also called "inter prediction" or "motion compensated prediction") predicts a current video block using reconstructed pixels from already coded video pictures. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically conveyed by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference.

[0025] Also, if multiple reference pictures are supported, one reference picture index is further sent and used to identify which reference picture in the reference picture storage the temporal prediction signal comes from. After spatial and / or temporal prediction, a mode decision block in the encoder chooses the best prediction mode, for example based on a rate-distortion optimization method. The prediction block is then subtracted from the current video block, and the prediction residual is decorrelated using a transform and quantized.

[0026] The quantized residual coefficients are inverse quantized and inverse transformed to form a reconstructed residual, which is then added back to the prediction block to form the reconstructed signal of the CU. Furthermore, in-loop filtering, such as a deblocking filter, sample adaptive offset (SAO), and adaptive in-loop filter (ALF), may be applied to the reconstructed CU, which is then put into a reference picture store and used to encode future video blocks. To form the output video bitstream, the coding mode (inter or intra), prediction mode information, motion information, and the quantized residual coefficients are all sent to an entropy coding unit for further compression and packing to form the bitstream.

[0027] 2 shows a schematic block diagram of a block-based video decoder. Specifically, FIG. 2 shows a block diagram of an exemplary decoder 200 according to this disclosure. The decoder 200 includes a bitstream 210, an entropy decoding 212, an inverse quantization 214, an inverse transform 216, an adder 218, an intra / inter mode selection 220, an intra prediction 222, a memory 230, an in-loop filter 228, a motion compensation 224, a picture buffer 226, prediction relationship information 234, and a video output 232.

[0028] In FIG. 2, the video bitstream is first entropy decoded in an entropy decoding unit. The coding mode and prediction information are sent to a spatial prediction unit (for intra-coding) or a temporal prediction unit (for inter-coding) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit and an inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then added together. The reconstructed block may further pass through an in-loop filter and then stored in a reference picture store. The reconstructed video in the reference picture store is then sent to drive a display device, as well as used to predict future video blocks.

[0029] Bidirectional Optical Flow

[0030] Traditional bidirectional prediction in video coding is a simple combination of two temporal prediction blocks obtained from an already reconstructed reference picture. However, due to the limitations of block-based motion compensation, there may remain a small amount of motion that can be observed between the samples of the two prediction blocks, thus reducing the efficiency of the motion-compensated prediction. In VVC, bidirectional optical flow (BDOF) is applied to reduce the effect of such motion on all samples within one block.

[0031] FIG. 4 illustrates an example of a bidirectional optical flow (BDOF) model according to the present disclosure. BDOF is a sample-by-sample motion compensation that is performed in addition to block-based motion compensation prediction when bidirectional prediction is used. The motion compensation for each 4×4 subblock (v x ,v y ) is calculated by minimizing the difference between the L0 and L1 predicted samples after BDOF is applied within one 6 × 6 window Ω around the subblock. Specifically, (v x ,v y ) is

[0032]

number

[0033] In the above formula,

[0034]

number

[0035] is a floor function, clip3(min,max,x) is a function that clips a given value x in the range [min,max], the symbol >> denotes a bitwise right shift operation, the symbol << denotes a bitwise left shift operation, and th BDOF is the motion compensation threshold to prevent propagation errors due to irregular local motion, and 2 13-BDwhere BD is the bit depth of the input video. In (1),

[0036]

number

[0037] It is.

[0038] S 1 , S 2 , S 3 , S 5 , and S 6 The value of is

[0039]

number

[0040] It is calculated as follows:

[0041]

number

[0042] In the above equation, I (k) (i,j) is the sample value of the predicted signal at coordinate (i,j) in list k, k=0,1, generated with medium precision (i.e., 16 bits);

[0043]

number

[0044] and

[0045]

number

[0046] are the horizontal and vertical gradients of a sample obtained by directly calculating the difference between its two adjacent samples, i.e.,

[0047]

number

[0048] It is.

[0049] Based on the motion correction derived in (1),

[0050]

number

[0051] The final bidirectional prediction samples of a CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the optical flow model, as shown by offset are the right shift and offset values ​​applied to combine the L0 and L1 prediction signals for bi-prediction, and are equal to 15-BD and 1<<(14-BD)+2·(1<<13), respectively. Table 1 shows the specific bit-widths of the intermediate parameters involved in the BDOF process. The bit-width is, for example, the number of bits required to represent a value. As shown in the table, the internal bit-width of the entire BDOF process does not exceed 32 bits. In addition, in (1), the multiplication with the worst possible input is performed with 15-bit and 4-bit inputs, v x S 2,m Therefore, a 15-bit multiplier is sufficient for BDOF.

[0052] [Table 1] JPEG0007586988000011.jpg92150

[0053] Efficiency of bidirectional predictive forecasting

[0054] Although BDOF can improve the efficiency of bi-predictive prediction, its design can still be further improved. Specifically, controlling the bit width of intermediate parameters in the existing BDOF design of VVC is revealed in this disclosure.

[0055] As shown in Table 1, the parameters θ(i,j) (i.e., the difference between the L0 and L1 prediction samples) and the parameters ψ x (i,j) and ψ y (i,j) (i.e., the sum of horizontal / vertical L0 and L1 gradient values) are represented with the same bit-width of 11 bits. Although such a method can facilitate the overall control of the internal bit-width for BDOF, it is suboptimal in terms of the accuracy of the derived motion compensation. This is because these gradient values ​​are calculated as the difference between adjacent predicted samples, as shown in (4), and the high-pass nature of such a process makes the derived gradients unreliable in the presence of noise, e.g., noise captured in the original video and coding noise generated during the encoding process. This means that it is not always beneficial to represent the gradient values ​​with a high bit-width.

[0056] As shown in Table 1, the maximum bit width usage of the entire BDOF process is the vertical motion compensation v y Here, first, S 6 (27 bits) is shifted left by 3 bits, then ((v x S 2,m )<<12+v x S 2,s ) / 2 (30 bits). Thus, the maximum bit width of the current design is equal to 31 bits. In practical hardware implementations, the encoding process with a maximum internal bit width larger than 16 bits is usually performed by a 32-bit implementation. Therefore, the existing design does not fully utilize the available dynamic range of the 32-bit implementation. This may lead to unnecessary precision loss of the BDOF derived motion compensation.

[0057] Improving the Efficiency of Bidirectional Prediction Using BDOF

[0058] In this disclosure, an improved bit-width control method is proposed to address two problems of the bit-width control method pointed out in the chapter "Efficiency of Bidirectional Prediction" on existing BDOF designs.

[0059] FIG. 5 illustrates a bitwidth control method for encoding a video signal according to the present disclosure.

[0060] In step 510, a first reference picture I associated with the sub-block is (0) and the second reference picture I (1) In display order, the first reference picture I (0) is a second reference picture I that precedes the current picture. (1) For example, a reference picture may be a video picture that is adjacent to the current picture being coded.

[0061] In step 512, the first reference picture I (0) The first predicted sample from the subblock to the reference block in (0) (i,j). For example, the first predicted sample I (0) (i,j) may be a prediction sample that uses a motion vector in the L0 list of the previous reference picture in display order.

[0062] In step 514, the second reference picture I (1) The second prediction sample I from the subblock to the reference block in (1) (i,j). For example, the second prediction sample I (1) (i,j) may be a predicted sample that uses a motion vector in the L1 list of the next reference picture in display order.

[0063] In step 516, the internal bit width of the bidirectional optical flow (BDOF) is controlled by varying the internal bit width of the intermediate parameter. The intermediate parameter is the first prediction sample I (0) (i,j) and the second prediction sample I (1) (i,j). The gradient values ​​include horizontal and vertical gradient values ​​derived based on the two adjacent predicted samples (the first predicted sample I (0) (i,j) and the second prediction sample I (1) (i,j) is the difference between

[0064] In step 518, the first prediction sample I (0) (i,j) and the second prediction sample I (1) (i,j) is used to obtain a motion compensation value based on the BDOF applied to the sub-block.

[0065] In step 520, bidirectional predicted samples of the sub-block are obtained based on the motion compensation value.

[0066] In one embodiment of the present disclosure, in order to overcome the negative effect of gradient estimation error, the gradient value in (4) is

[0067]

number

[0068] and

[0069]

number

[0070] In order to reduce the internal bit width of the gradient value, an additional right shift n grad Specifically, the horizontal and vertical gradients at each sample location are

[0071]

number

[0072] It is calculated as:

[0073] Furthermore, in order to control the entire BDOF process so that it can be operated at an appropriate internal bit width,

[0074]

number

[0075] As shown, the variable ψ x (i,j), ψ y (i,j), and θ(i,j) are calculated using an additional bit shift n adj will be introduced.

[0076] In Table 2, the parameter ψ is compared to the existing BDOF design in Table 1, where the three parameters are represented in the same dynamic range (i.e., 21 bits) due to the modification of the number of right-shifted bits applied in (6) and (7). x (i,j), ψ y The dynamic ranges of θ(i,j) and θ(i,j) are different. Such variations are caused by the internal parameter S 1 , S 2 , S 3 , S 5 , and S 6 This may increase the bit width of S, which may potentially increase the maximum bit width of the internal BDOF process beyond 32 bits. Therefore, to ensure a 32-bit implementation, 2 and S 6 Two additional clipping operations are introduced when calculating the value of

[0077]

number

[0078] In the above formula, B 2 and B. 6 are S 2 and S 6 is a parameter for controlling the output dynamic range of (8). For example, bit depth represents the number of bits used to define each pixel. Note that unlike the gradient calculation, the clipping operation in (8) is applied only once to calculate the motion compensation for each 4 × 4 sub-block in a CU of one BDOF, i.e., it is invoked on a 4 × 4 unit basis. Therefore, the corresponding complexity increase due to the clipping operation introduced in the proposed method is very small.

[0079] In practice, n is chosen to achieve different trade-offs between the intermediate bit width and the accuracy of the internal BDOF derivation. grad , n adj , B 2 , and B. 6 In one embodiment of the present disclosure, different values ​​of n grad and n adj to 2, B 2 to 25, B 6 It is proposed to set B to 27. 2 to 26, B 6 It is proposed to set n to 28. grad and n adj It is suggested to set it to 6, 1, or 4.

[0080] Table 2 shows the corresponding bit-width of each intermediate parameter when the proposed bit-width control method is applied to BDOF. In Table 2, grey highlights the changes applied with the proposed bit-width control method when compared with the existing BDOF design of VVC (shown in Table 1). As can be seen in Table 2, for the proposed bit-width control method, the internal bit-width of the whole BDOF process does not exceed 32 bits. In addition, with the proposed design, the maximum bit-width is just 32 bits, which can fully utilize the available dynamic range of the 32-bit hardware implementation. On the other hand, as shown in this table, the multiplication with the worst possible input is v x S 2,m The input S 2,m is 14 bits, and the input v x is 6 bits. Therefore, similar to the existing BDOF design, one 16-bit multiplier is large enough when the proposed method is applied.

[0081] [Table 2] JPEG0007586988000018.jpg125150

[0082] In the above method, v x and v y When deriving , a clipping operation is added to avoid overflow of intermediate parameters, as shown in equation (8). However, such clipping is only needed when the correlation parameters are accumulated within a large local window. When one small window is applied, overflow cannot occur. Therefore, in another embodiment of the present disclosure, the following bit depth control method is proposed for the BDOF method without clipping, as described below.

[0083] First, the gradient value of (4) at each sample position

[0084]

number

[0085] and

[0086]

number

[0087] but,

[0088]

number

[0089] It is calculated as:

[0090] Second, the correlation parameter ψ which is then used in the BDOF process x (i,j), ψ y (i,j), and θ(i,j) are

[0091]

number

[0092] It is calculated as:

[0093] Third, S 1 , S 2 , S 3 , S 5 , and S 6 The value of

[0094]

number

[0095] It is calculated as:

[0096] Fourth, the motion compensation (v x ,v y )but,

[0097]

number

[0098] It is derived as:

[0099] Fifth,

[0100]

number

[0101] As shown by, the final bidirectional prediction samples of a CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the optical flow model.

[0102] FIG. 6 illustrates an exemplary BDOF bitwidth control method according to this disclosure.

[0103] In step 610, a first reference picture I associated with the sub-block is (0) and the second reference picture I (1) In display order, the first reference picture I (0) is a second reference picture I (1) is after the current picture.

[0104] In step 612, the first reference picture I (0) The first predicted sample from the subblock to the reference block in (0) Get (i,j).

[0105] In step 614, the second reference picture I (1) The second prediction sample I from the subblock to the reference block in (1) Get (i,j).

[0106] In step 616, the internal bit width of the bidirectional optical flow (BDOF) is controlled by varying the internal bit width of the derived gradient values. (0) (i,j) and the second prediction sample I (1) Includes horizontal and vertical gradient values ​​derived based on (i,j).

[0107] In step 618, the first prediction sample I (0) (i,j) and the second prediction sample I (1) Obtain a motion compensation value based on the BDOF and a local window applied to the subblock using (i,j), where the local window contains the subblock and has a window size of less than 6x6 pixels.

[0108] In step 620, bi-directional predicted samples of the sub-block are obtained based on the motion compensation value.

[0109] 7 illustrates a computing environment 710 coupled to a user interface 760. The computing environment 710 may be part of a data processing server. The computing environment 710 includes a processor 720, a memory 740, and an I / O interface 750.

[0110] The processor 720 typically controls the overall operation of the computing environment 710, such as operations related to display, data acquisition, data communication, and image processing. The processor 720 may include one or more processors to execute instructions for performing all or some of the steps of the methods described above. Additionally, the processor 720 may include one or more modules that facilitate interaction between the processor 720 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single chip machine, a GPU, or the like.

[0111] The memory 740 is configured to store various types of data to support the operation of the computing environment 710. Examples of such data include instructions for any application or method operated on the computing environment 710, video data sets, image data, etc. The memory 740 may be implemented by using any type of volatile or non-volatile memory device, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks, or a combination thereof.

[0112] The I / O interface 750 provides an interface between the processor 720 and a peripheral interface module, such as a keyboard, a click wheel, buttons, etc. The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 750 may be coupled to an encoder and a decoder.

[0113] In one embodiment, a non-transitory computer readable storage medium is also provided that comprises a number of programs, such as those comprised in memory 740, that are executable by processor 720 in computing environment 710 to perform the methods described above. For example, the non-transitory computer readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0114] A non-transitory computer-readable storage medium stores a plurality of programs for execution by a computing device having one or more processors, the plurality of programs, when executed by the one or more processors, causing the computing device to perform the above-described method for bitwidth control for BDOF.

[0115] In one embodiment, the computing environment 710 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), graphical processing units (GPUs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

Claims

1. A bidirectional optical flow (BDOF) bitwidth control method for decoding a video signal, comprising: obtaining a first reference picture and a second reference picture associated with a video sub-block, the first reference picture being before a current picture and the second reference picture being after the current picture in display order; A first prediction sample I for a sample of the video sub-block from the first reference picture. (0) and A second prediction sample I for the sample of the video sub-block from the second reference picture. (1) and First prediction sample I (0) (i+1,j) and the first predicted sample I (0) (i-1, j), where i and j correspond to coordinates of the one sample; and obtaining a first horizontal gradient for the one sample of the samples based on (i-1, j), where i and j correspond to coordinates of the one sample. Second prediction sample I (1) (i+1,j) and the second predicted sample I (1) (i-1, j) to obtain a second horizontal gradient for the one sample; First prediction sample I (0) (i, j+1) and the first predicted sample I (0) (i, j−1), obtaining a first vertical gradient for the one sample; Second prediction sample I (1) (i, j+1) and the second predicted sample I (1) (i, j−1) to obtain a second vertical gradient for the one sample; and obtaining motion compensation for the one sample of the video sub-block by controlling a bit width of intermediate parameters, the intermediate parameters being the first and second horizontal gradients, the first and second vertical gradients, and a first prediction sample I. (0) (i,j) and the second predicted sample I (1) (i,j) based sample difference; obtaining a bi-directional predictive sample for the one sample of the video sub-block based on the motion compensation, a difference between the first horizontal gradient and the second horizontal gradient, and a difference between the first vertical gradient and the second vertical gradient; obtaining motion compensation for the one sample of the video sub-block by controlling a bit-width of the intermediate parameter; performing a right shift operation using a first bit shift value to control bit widths of the first and second horizontal gradients; performing a right shift operation using the first bit shift value to control a bit width of the first and second vertical gradients; obtaining a first correlation value, the first correlation value being a sum of the first horizontal gradient and the second horizontal gradient; obtaining a second correlation value, the second correlation value being a sum of the first vertical gradient and the second vertical gradient; right-shifting the first correlation value using a second bit-shift value to obtain a modified first correlation value; and right-shifting the second correlation value using a second bit-shift value to obtain a modified second correlation value.

2. Obtaining motion compensation for the one sample of the video sub-block by controlling a bit-width of the intermediate parameter includes, for the one sample of the video sub-block, A first predicted sample I for the one sample is obtained using a third bit shift value. (0) obtaining a first modified prediction sample by right-shifting (i, j); A second predicted sample I for the one sample is calculated using the third bit shift value. (1) obtaining a second modified prediction sample by right-shifting (i, j); obtaining the sample difference for the one sample, the sample difference being a difference between the first modified predicted sample and the second modified predicted sample; The method of claim 1 further comprising:

3. 3. The method of claim 2, wherein the second bit shift value and the third bit shift value are equal to 1 and 4, respectively. obtaining a first internal sum value based on a sum of the first correlation value and the second correlation value in the video sub-block; clipping the first internal sum based on a first parameter, the first parameter controlling an output bit depth of the first internal sum; obtaining a second internal sum value based on the second correlation value and the sum of sample differences within the video sub-block; clipping the second internal sum value based on a second parameter, the second parameter controlling an output bit depth of the second internal sum value; and The method of claim 2 , further comprising:

5. The method of claim 4 , wherein the first parameter is equal to 26 and the second parameter is equal to 28.

6. A bidirectional optical flow (BDOF) bitwidth control method for decoding a video signal, comprising: obtaining a first reference picture and a second reference picture associated with a 4x4 video sub-block, the first reference picture being before a current picture and the second reference picture being after the current picture in display order; A first prediction sample I for the sample of the 4×4 video sub-block from the first reference picture. (0) and A second prediction sample I for the sample of the 4×4 video sub-block from the second reference picture. (1) and and obtaining motion compensation for samples of the 4×4 video sub-block based on a 6×6 window applied to the 4×4 video sub-block by controlling a bit width of an intermediate parameter, the intermediate parameter being a first prediction sample I (0) and the second prediction sample I (1) and obtaining at least one of a horizontal gradient, a vertical gradient, and a sample difference derived based on and obtaining bi-predictive samples of samples of the 4×4 video sub-block based on the motion compensation. method.

7. obtaining motion compensation for samples of the video sub-block based on a 6×6 window applied to the 4×4 video sub-block by controlling a bit-width of the intermediate parameters; performing a right shift operation using a first bit shift value to control a bit width of the horizontal gradient; 7. The method of claim 6, further comprising controlling a bit-width of the vertical gradient by performing a right shift operation using the first bit-shift value.

8. Obtaining motion compensation for a sample of the 4×4 video sub-block based on a 6×6 window applied to the 4×4 video sub-block by controlling a bit width of the intermediate parameter includes, for one sample of the 4×4 video sub-block: obtaining a first correlation value, the first correlation value being a function of the first predicted sample I for the one sample; (0) and the second prediction sample I (1) and obtaining, which is the sum of the horizontal gradients of obtaining a second correlation value, the second correlation value being a function of the first predicted sample I for the one sample; (0) and the second prediction sample I (1) and obtaining, obtaining a modified first correlation value by right-shifting the first correlation value by one; 7. The method of claim 6, further comprising: obtaining a modified second correlation value by right-shifting the second correlation value by one.

9. Obtaining motion compensation for a sample of the 4×4 video sub-block based on a 6×6 window applied to the 4×4 video sub-block by controlling a bit width of the intermediate parameter includes, for one sample of the 4×4 video sub-block: The first predicted sample I for the one sample (0) obtaining a first modified prediction sample by right-shifting (i,j) by 4; A second predicted sample I for the one sample (1) obtaining a second modified prediction sample by right-shifting (i,j) by 4; obtaining a sample difference for the one sample, the sample difference being a difference between the first modified predicted sample and the second modified predicted sample; The method of claim 8 further comprising:

10. one or more processors; a memory storing instructions for decoding a video signal; A computing device comprising: The instructions, when executed by the one or more processors, cause the computing device to perform a method according to any of claims 1 to 5.

11. one or more processors; a memory storing instructions for decoding a video signal; A computing device comprising: A computing device, the instructions, when executed by the one or more processors, causing the computing device to perform a method according to any of claims 6 to 9.

12. A non-transitory computer-readable storage medium storing instructions for decoding a video signal, the instructions, when executed by a processor, causing the processor to perform a method according to any one of claims 1 to 5.

13. A non-transitory computer-readable storage medium storing instructions for decoding a video signal, the instructions, when executed by a processor, causing the processor to perform a method according to any one of claims 6 to 9.

14. A computer program causing a computer to carry out the method according to any one of claims 1 to 5 or any one of claims 6 to 9.

15. A bidirectional optical flow (BDOF) bitwidth control method for encoding a video signal, comprising: determining a first reference picture and a second reference picture associated with a video sub-block, the first reference picture being before a current picture and the second reference picture being after the current picture in display order; A first prediction sample I for a sample of the video sub-block from the first reference picture. (0) and determining A second prediction sample I for the sample of the video sub-block from the second reference picture. (1) and determining First prediction sample I (0) (i+1,j) and the first predicted sample I (0) (i-1, j), where i and j correspond to coordinates of the one sample; and determining a first horizontal gradient for the one sample of the samples based on (i-1, j), where i and j correspond to coordinates of the one sample. Second prediction sample I (1) (i+1,j) and the second predicted sample I (1) determining a second horizontal gradient for the one sample based on (i-1, j); First prediction sample I (0) (i, j+1) and the first predicted sample I (0) determining a first vertical gradient for the one sample based on (i, j−1); Second prediction sample I (1) (i, j+1) and the second predicted sample I (1) determining a second vertical gradient for the one sample based on (i, j−1); determining a motion compensation for the one sample of the video sub-block by controlling a bit width of an intermediate parameter, the intermediate parameter being a first and second horizontal gradient, a first and second vertical gradient, and a first prediction sample I; (0) (i,j) and the second predicted sample I (1) a sample difference based on (i,j); determining a bi-directionally predicted sample for the one sample of the video sub-block based on the motion compensation, a difference between the first horizontal gradient and the second horizontal gradient, and a difference between the first vertical gradient and the second vertical gradient; determining a motion compensation for the one sample of the video sub-block by controlling a bit-width of the intermediate parameter; performing a right shift operation using a first bit shift value to control bit widths of the first and second horizontal gradients; performing a right shift operation using the first bit shift value to control a bit width of the first and second vertical gradients; determining a first correlation value, the first correlation value being a sum of the first horizontal gradient and the second horizontal gradient; determining a second correlation value, the second correlation value being a sum of the first vertical gradient and the second vertical gradient; determining a modified first correlation value by right-shifting the first correlation value using a second bit-shift value; determining a modified second correlation value by right shifting the second correlation value using a second bit shift value.

16. Determining a motion compensation for the one sample of the video sub-block by controlling a bit-width of the intermediate parameter includes, for the one sample of the video sub-block, A first predicted sample I for the one sample is obtained using a third bit shift value. (0) determining a first modified prediction sample by right-shifting (i,j); A second predicted sample I for the one sample is calculated using the third bit shift value. (1) determining a second modified prediction sample by right-shifting (i,j); 16. The method of claim 15, further comprising: determining the sample difference for the one sample, the sample difference being a difference between the first modified predicted sample and the second modified predicted sample.

17. 17. The method of claim 16, wherein the second bit shift value and the third bit shift value are equal to 1 and 4, respectively.

18. determining a first internal sum value based on a sum of the first correlation value and the second correlation value within the video sub-block; clipping the first internal sum based on a first parameter, the first parameter controlling an output bit depth of the first internal sum; determining a second internal sum value based on the second correlation value and the sum of sample differences within the video sub-block; clipping the second internal sum value based on a second parameter, the second parameter controlling an output bit depth of the second internal sum value; and 20. The method of claim 16, further comprising:

19. The method of claim 18 , wherein the first parameter is equal to 26 and the second parameter is equal to 28.

20. A bidirectional optical flow (BDOF) bitwidth control method for encoding a video signal, comprising: determining a first reference picture and a second reference picture associated with a 4x4 video sub-block, the first reference picture being before a current picture and the second reference picture being after the current picture in display order; A first prediction sample I for the sample of the 4×4 video sub-block from the first reference picture. (0) and determining A second prediction sample I for the sample of the 4×4 video sub-block from the second reference picture. (1) and determining determining a motion compensation for samples of the 4×4 video sub-block based on a 6×6 window applied to the 4×4 video sub-block by controlling a bit width of an intermediate parameter, the intermediate parameter being a first prediction sample I (0) and the second prediction sample I (1) determining at least one of a horizontal gradient, a vertical gradient, and a sample difference based on determining bi-predictive samples of samples of the 4x4 video sub-block based on the motion compensation. method.

21. determining motion compensation for samples of the video sub-block based on a 6×6 window applied to the 4×4 video sub-block by controlling a bit-width of the intermediate parameter; performing a right shift operation using a first bit shift value to control a bit width of the horizontal gradient; 21. The method of claim 20, further comprising controlling a bit-width of the vertical gradient by performing a right shift operation using the first bit-shift value.

22. determining a motion compensation for a sample of the 4×4 video sub-block based on a 6×6 window applied to the 4×4 video sub-block by controlling a bit-width of the intermediate parameter, for one sample of the 4×4 video sub-block: determining a first correlation value, the first correlation value being a function of the first predicted sample I for the one sample; (0) and the second prediction sample I (1) determining the sum of the horizontal slopes of determining a second correlation value, the second correlation value being a function of the first predicted sample I for the one sample; (0) and the second prediction sample I (1) determining the sum of the vertical gradients of determining a modified first correlation value by right-shifting the first correlation value by one; 21. The method of claim 20, further comprising determining a modified second correlation value by right shifting the second correlation value by one.

23. determining a motion compensation for a sample of the 4×4 video sub-block based on a 6×6 window applied to the 4×4 video sub-block by controlling a bit-width of the intermediate parameter, for one sample of the 4×4 video sub-block: The first predicted sample I for the one sample (0) determining a first modified prediction sample by right-shifting (i,j) by 4; A second predicted sample I for the one sample (1) determining a second modified prediction sample by right-shifting (i,j) by 4; determining a sample difference for the one sample, the sample difference being a difference between the first modified predicted sample and the second modified predicted sample; 23. The method of claim 22, further comprising:

24. one or more processors; a memory storing instructions for encoding a video signal; A computing device comprising: The instructions, when executed by the one or more processors, cause the computing device to perform a method according to any of claims 15 to 19.

25. one or more processors; a memory storing instructions for encoding a video signal; A computing device comprising: A computing device, the instructions, when executed by the one or more processors, causing the computing device to perform a method according to any of claims 20 to 23.

26. A non-transitory computer readable storage medium storing instructions for encoding a video signal, the instructions, when executed by a processor, causing the processor to perform a method according to any one of claims 15 to 19.

27. A non-transitory computer readable storage medium storing instructions for encoding a video signal, the instructions, when executed by a processor, causing the processor to perform a method according to any one of claims 20 to 23.

28. A computer program causing a computer to carry out the method according to any one of claims 15 to 19 or any one of claims 20 to 23.

29. Carrying out the method according to claims 15 to 23 to generate a bitstream; storing the bitstream in a non-transitory computer-readable storage medium.

30. A method for transmitting a bitstream, the bitstream being generated by a method according to any one of claims 15 to 19 or claims 20 to 23.

Citation Information

Patent Citations

  • Moving image decoding device, moving image encoding device, and prediction image generation device

    WO2017134957A1

  • Inter prediction refinement based on BI-directional optical flow (BIO)

    WO2018169989A1

  • Motion-compensation prediction based on BI-directional optical flow

    WO2019010156A1

  • Complexity reduction and bit-width control for BI-directional optical flow

    WO2020061082A1