Bit width control against bidirectional optical flow
By controlling the internal bit width of gradient values and motion correction parameters in BDOF, the inefficiencies and inaccuracies in existing BDOF designs are addressed, resulting in improved video encoding efficiency and accuracy.
Patent Information
- Application Number
- JP2024154127
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-01-06
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-01-06
AI Technical Summary
Existing bidirectional optical flow (BDOF) designs in video coding face inefficiencies due to high internal bit widths and inaccuracies in gradient estimation, leading to suboptimal motion compensation and increased computational complexity.
Implementing additional right shifts to control the internal bit width of gradient values and motion correction parameters in the BDOF process, ensuring accurate motion compensation while optimizing hardware implementation.
Enhances the efficiency and accuracy of bidirectional optical flow by reducing bit width constraints and utilizing the full dynamic range of hardware implementations, improving video encoding performance.
Smart Images

Figure 2024161272000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 62 / 788,898, filed January 6, 2019. The entire disclosure of the above application is hereby incorporated by reference in its entirety. There are.
[0002] This disclosure generally relates to encoding (e.g., coding and decoding) video data. More specifically, this disclosure provides a bidirectional optical flow (BDOF) method for video coding. In certain exemplary embodiments, the present invention relates to a method and apparatus for improving a BDOF (backward axial deflection) processing method. is used to find a reference picture that occurs in the opposite direction to the current picture (e.g., the first reference picture having a display order before the current picture, and a display order after the current picture The method of predicting a block bidirectionally from a first reference picture (a second reference picture having a first reference picture with a second reference picture having a second reference picture with ... can be. [Background technology]
[0003] This section provides background information related to the present disclosure. The information contained in this section is not necessarily conventional. It should not necessarily be interpreted as a technique.
[0004] Various video encoding techniques can be used to compress the video data. The video encoding is performed according to one or more video encoding standards. Coding standards include Versatile Video Coding (VVC), Joint Exploration Model Coding (JEM ), High Efficiency Video Coding (H.265 / HEVC), Advanced Video Coding (H.264 / A VC), Moving Picture Experts Group Coding (MPEG), etc. Video coding generally exploits redundancy present in a video image or sequence. A prediction method (e.g., inter-prediction, intra-prediction, etc.) is utilized. The key goal of is to achieve lower bitrates while avoiding or minimizing degradation of video quality. The goal of this paper is to compress video data into a format that uses
[0005] The first version of the HEVC standard was finalized in October 2013. Compared to the previous generation video coding standard H.264 / MPEG AVC, it uses approximately 50% less bit Offers rate savings or equivalent perceptual quality. The HEVC standard offers significant improvements over its predecessor. It offers significant coding improvements, but the additional coding tools give it better coding than HEVC. It has been proven that it is possible to achieve high efficiency. Both PEGs are working on exploring new coding techniques for future video coding standards. In October 2015, the company announced that it had developed an advanced technology that could enable substantial improvements in coding efficiency. To ITU-T VECG and ISO / IEC MPEG to initiate significant research Therefore, a joint video exploration team (JVET) was formed. JVET is a By integrating several additional coding tools in addition to the test model (HM), We developed a reference software called the Joint Exploration Model (JEM).
[0006] In October 2017, ITU-T and ISO / IEC agreed to go beyond HEVC. The two companies announced a joint Call for Proposals (CfP) on video compression for In April 2018, 23 CfP responses were accepted and evaluated at the 10th JVET Conference. It was demonstrated that the compression efficiency was improved by about 40% compared to VC. Based on the evaluation results, JVET is developing a new generation video coding standard called Versatile Video Coding (VVC). In the same month, the company launched a new project to demonstrate a reference implementation of the VVC standard. In 2013, a single reference software code base was established called the VVC Test Model (VTM). It was done. Summary of the Invention
[0007] This section provides an overview of the disclosure and does not necessarily represent the complete scope or entirety of the disclosure. It is not a comprehensive disclosure of all features.
[0008] According to a first aspect of the present disclosure, there is provided a method for detecting a plurality of processors, the method comprising: and a memory for storing a plurality of programs to be executed by the processor. It runs on a single operating device and uses bidirectional optical flow (BDO) for video coding. F) A method of controlling the bit width for the first reference picture and the second reference picture The second reference picture is different from the first reference picture and is a first reference picture. The first reference picture is displayed before the current picture, and the second reference picture is displayed after the current picture. The first block of the first reference picture is displayed after the current block of the current picture. A first motion vector (MV0) is determined from the current block to the prediction L0 of L1. A second motion vector (MV1) to a second prediction L1 of the current reference picture is determined. A first temporal distance (τ0) from the current picture to the first reference picture is determined, A second temporal distance (τ1) from the first prediction L0 to a second reference picture is determined. The motion compensation is calculated by minimizing the difference between the first prediction L1 and the second prediction L2. A gradient value and a second gradient value are calculated, and an internal bit width of the first gradient value and the second gradient value is An additional right shift is introduced to reduce the first and second gradient values The sum is calculated and an additional right shift is introduced into this sum to reduce its bit width. The difference between the first predicted sample L0 and the second predicted sample L1 is calculated, and In order to reduce the bit width, the first predicted sample L0 and the second predicted sample L1 are An additional right shift is introduced.
[0009] According to a second aspect of the present disclosure, a computing device includes one or more processors. a non-transitory storage device coupled to the one or more processors; and a plurality of programs stored in the device. The plurality of programs may be one or more programs. When executed by the processor, the computing device and a second reference picture, the second reference picture being different from the first reference picture. The first reference picture is displayed before the current picture, and the second reference picture is displayed before the current picture. It is displayed after the current picture. It is the first reference picture from the current block of the current picture. A first motion vector (MV0) is determined for the first prediction L0 of the current block. A second motion vector (MV1) is determined from the second prediction L1 of the second reference picture. A first temporal distance (τ0) from the current picture to the first reference picture is determined. A second temporal distance (τ1) from the current picture to the second reference picture is determined. The motion compensation is calculated by minimizing the difference between the prediction L0 and the second prediction L1. A first gradient value and a second gradient value are calculated, and an internal of the first gradient value and the second gradient value is calculated. To reduce the bit width, an additional right shift is introduced. The sum of the gradient values is calculated and an additional right shift is introduced into this sum to reduce its bit width. The difference between the first predicted sample L0 and the second predicted sample L1 is calculated, In order to reduce the internal bit width of the difference between the first predicted sample L0 and the second predicted sample L1, An additional right shift is introduced in pull L1.
[0010] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium includes one or more A plurality of programs for execution by a computing device having a processor of The programs, when executed by one or more processors, , decoding the first reference picture and the second reference picture on a computing device. The second reference picture is different from the first reference picture, and the first reference picture is The first reference picture is displayed before the current picture, and the second reference picture is displayed after the current picture. The first prediction L0 of the first reference picture from the current block of the current picture is The first motion vector (MV0) from the current block to the second reference picture is determined. A second motion vector (MV1) is determined from the current picture to the second prediction L1. A first temporal distance (τ0) from the current picture to the second reference picture is determined. A second temporal distance (τ1) to the picture is determined. The motion compensation is calculated by minimizing the difference between the first gradient value and the second gradient value. A gradient value is calculated, and an internal bit width of the first gradient value and the second gradient value is reduced. , an additional right shift is introduced. The sum of the first gradient value and the second gradient value is calculated, To reduce the bit width, an additional right shift is introduced in this sum. The difference between the first predicted sample L0 and the second predicted sample L1 is calculated, and the internal bit width of this difference is reduced. In order to achieve this, an additional right shift is applied to the first predicted sample L0 and the second predicted sample L1. It will be introduced.
[0011] A set of exemplary, non-limiting embodiments of the present disclosure will now be described in conjunction with the accompanying drawings. Those skilled in the relevant art will be able to easily understand the structures, methods, or or functional variations may be implemented, and all such variations are within the scope of this disclosure. Where no inconsistency exists, the teachings of different embodiments may, but need not, be incorporated into one another. They can also be combined. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a block diagram of an exemplary block-based hybrid video coding system according to an exemplary embodiment including an exemplary encoder suitable for use with a bidirectional optical flow (BDOF) method. [Diagram 2] FIG. 2 is a block diagram of an exemplary decoder according to an exemplary embodiment suitable for use with a bidirectional optical flow (BDOF) method. [Diagram 3]FIG. 3A is a diagram showing an exemplary distribution type according to an exemplary embodiment, namely, a 4-way split; FIG. 3B is a diagram showing an exemplary distribution type according to an exemplary embodiment, namely, a horizontal 2-way split; FIG. 3C is a diagram showing an exemplary distribution type according to an exemplary embodiment, namely, a vertical 2-way split; FIG. 3D is a diagram showing an exemplary distribution type according to an exemplary embodiment, namely, a horizontal 3-way split; and FIG. 3E is a diagram showing an exemplary distribution type according to an exemplary embodiment, namely, a vertical 3-way split. [Figure 4] FIG. 4 is a diagram illustrating an exemplary bidirectional optical flow (BDOF) model. [Diagram 5] FIG. 5 is a flowchart of an example method for predicting a block of video data according to an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] The terms used in this disclosure are not intended to limit the disclosure but to illustrate specific examples. Unless otherwise clearly implied by context, the present disclosure and the appended claims As used in the scope, the singular forms "a," "an," and "the" also refer to the plural. In this document, the term "and / or" means any and all of the relevant items in a list. It should be understood that the term refers to any possible combination.
[0014] In this specification, terms such as "first," "second," and "third" are used to describe various pieces of information. Any terminology may be used, but this information is not limited by these terms. It should be understood that these terms do not distinguish one information category from another. For example, the first The information can be referred to as the second information, and similarly the second information can be referred to as the first information. In this specification, the term "if" can be used interchangeably with "when", depending on the context. )" or "upon" or "in response to" o)".
[0015] Throughout this specification, the terms "one embodiment," "embodiment," "alternative embodiment," "other embodiment," "particular ... References to an "embodiment" or the like include references to one or more particular features described in connection with an embodiment. It means that the structure or feature is included in at least one embodiment of the present disclosure. Thus, the references "in one embodiment" or "in an embodiment" in various places throughout this specification The references to phrases such as "in," "in an embodiment," or "in another embodiment" do not necessarily Moreover, all references in one or more embodiments are not necessarily to the same embodiment. The particular features, structures, or characteristics may be combined in any suitable manner.
[0016] This disclosure generally relates to encoding (e.g., coding and decoding) video data. More specifically, this disclosure provides a bidirectional optical flow (BDOF) method for video coding. In certain exemplary embodiments, the present invention relates to a method and apparatus for improving a BDOF (backward axial deflection) processing method. is used to find a reference picture that occurs in the opposite direction to the current picture (e.g., the first reference picture having a display order before the current picture, and a display order after the current picture The method of predicting a block bidirectionally from a first reference picture (a second reference picture having a first reference picture with a second reference picture having a second reference picture with ... can be.
[0017] The methods described herein relate to improving or enhancing BDOF. BDOF can be applied to the current block, which is a pixel-by-pixel motion vector for the current block. This is used to correct the motion vector and therefore the corresponding offset applied to the motion vector. The set value is used to predict the pixels of the current block. OF uses the motion vector to find a reference block in the opposite prediction direction to the current block. Interpolates blocks of video data in the current picture against reference blocks in the picture. Video coding methods reduce the redundancy inherent in video sequences. Spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or eliminate In the case of block-based video coding, a video slice (e.g. , a video frame or a portion of a video frame) into video blocks. When a video block is encoded, it is divided into tree blocks, coding units (CUs), and / or coding nodes. Spatial prediction is performed with respect to reference samples of neighboring blocks in the same picture. The method codes video blocks in an intra-coded (I) slice of a picture using the Video blocks in an inter-coded (P or B) slice of a picture The prediction is based on spatial prediction with respect to reference samples of neighboring blocks in the same picture, or on other reference samples. Temporal prediction can be used with respect to reference samples in the frame. , and the reference picture can be called a reference frame.
[0018] The spatial or temporal prediction results in a predicted block for coding the block. The residual data represents pixel differences between the original block to be coded and the predicted block. The center coding block is a motion vector that points to a block of reference samples that form the prediction block. and residual data indicating the difference between the coded block and the predicted block. Intra-coded blocks are coded according to the intra-coding mode and the residual data. For further compression, the residual data is converted from the pixel domain to the transform domain. The resulting residual transform coefficients can then be The transform coefficients can be quantized. The quantized The resulting transform coefficients can be scanned to produce a one-dimensional vector of transform coefficients, Encoding can be applied to achieve further compression.
[0019] For each inter-predicted block, one set of motion information may be available. The set of motion information may include motion information for forward and backward prediction directions. In this specification, the forward or first prediction direction and the backward or second prediction direction are referred to as bidirectional. The two prediction directions for the directional prediction mode are "forward", "first", "backward", and " The term "second" does not necessarily have a geometric meaning. The term generally refers to whether the reference picture appears before the current picture ("backwards") or In an exemplary embodiment, ,The forward prediction direction is the reference picture list 0 of the current picture (RefPicList0 ), and the backward prediction direction can correspond to reference picture list 1 (RefPicList It can respond to st1).
[0020] For each prediction direction, the motion information includes a reference index and a motion vector. In this form, the reference index is the current reference picture list (RefPicList0 or Re fPicList1) can be used to identify a reference picture. A vector has a horizontal component (x) and a vertical component (y). Generally, the horizontal component is To determine the x-coordinate of the lock, a reference to the position of the current block in the current picture is used. indicates the horizontal displacement in the reference picture, and the vertical component identifies the y coordinate of the reference block. Indicates the vertical displacement in the reference picture relative to the position of the current block.
[0021] Video coding standards use picture order counting to identify the display order of pictures. In some situations, the POC (Profile of Constraints) value is used for one coded video sequence. Two pictures in a sequence may have the same POC value, but this is typically due to does not occur in a coded video sequence. Therefore, the POC value of a picture is generally A single bit is unique and therefore can uniquely identify the corresponding picture. When multiple coded video sequences are present in a stream, they may have the same POC value. Pictures that correspond to each other may be close to each other in terms of decoding order. Typically, it involves constructing a reference picture list, deriving a reference picture set similar to HEVC, and It is used for scaling the motion vectors.
[0022] Conceptually, many video coding standards, including those already mentioned in the background chapter, are similar. For example, virtually all video coding standards use block-based It uses clock-based processing and shares a similar video encoding block diagram.
[0023] Like HEVC, VVC is structured as a block-based hybrid video coding framework. FIG. 1 illustrates a method for bidirectional optical flow according to an exemplary embodiment. A block-based hybrid video coding system including an exemplary encoder 100 that can The input video signal is processed in blocks (CUs). In M-1.0, a CU can be up to 128x128 pixels. However, Unlike HEVC, which splits blocks based on trees only, VVC uses quad-tree / binary-tree In order to adapt to local features that vary based on the / 3-ary tree, one coding tree unit ( In addition, the HEVC has multiple partition unit types. The concept of CU, prediction unit (PU), and transformation unit (TU) separation is eliminated. In VC, instead, each CU is always predicted and transformed without further partitioning. In a multi-type tree structure, one CTU is used as the base unit for both. The tree is divided into quadtrees, and then the leaf nodes of each quadtree are divided into binary and ternary trees. Therefore, it can be further divided.
[0024] Referring to FIG. 1, in an encoder 100, a video frame is prepared for processing in multiple For each given video block, an inter prediction technique or In the inter prediction, a prediction is formed based on the previously reconstructed Based on pixels from the captured frames, motion estimation and compensation are performed to generate one or more In intra prediction, a predictor is formed based on the reconstructed pixels in the current frame. A predictor is formed. Depending on the mode decision, the best predictor is chosen to predict the current block. You can select the child.
[0025] A prediction residual, which represents the difference between the current video block and its predictor, is sent to the transform circuit 102. In this specification, the term "circuitry" refers to hardware and the hardware that operates the hardware. The transform circuit 102 includes software for performing a discrete cosine transform (DCT) or applies a transform, such as a conceptually similar transform, to the residual block and returns a video Instead of DCT, wavelet transform, integer transform, subband A transform, discrete sine transform (DST), or other types of transforms can also be used. The transform circuit 102 applies a transform to the residual block to produce a set of transform coefficients. Residual information from the pixel domain can be transformed into a transform domain such as the frequency domain. do.
[0026] The resulting transform coefficients are then passed through transform circuit 102 for entropy reduction. The quantization circuit 104 then quantizes the input signal to further reduce the bit rate. The quantization process reduces some or all of these coefficients to a certain degree. The bit depth associated with the quantization can be reduced by adjusting the quantization parameter. The quantized coefficients can then be modified by The resulting stream is sent to an entropy coding circuit 106 to generate the entropy stream. The coding circuit 106 entropy codes the quantized transform coefficients. The tropy coding circuit 106 is a context-adaptive variable length coding (CAVLC) circuit. Context-adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another Context-based entropy coding techniques can be implemented. In this case, the context can be based on neighboring blocks. Following entropy coding by 6, the coded bitstream is then sent to another device. The video signal may then be transmitted to a device (e.g., a video decoder) for later transmission or retrieval. As shown in FIG. 1, the video block division information, the motion vector, Inter prediction circuitry and / or The prediction relationship information 110 from the intra prediction circuit 112 is also input to the entropy coding circuit 1 06 and stored in the compressed video bitstream 114.
[0027] The encoder 100 also requires decoder-related circuitry to reconstruct pixels for prediction purposes. First, the prediction residual is converted by the inverse quantization circuit 116 and the inverse transform circuit 118. This reconstructed prediction residual is combined with the block predictor 120 to produce the current quantization The circuit 116 and the inverse transform circuit 118 apply inverse quantization and inverse transform, respectively, to produce a pixel In a particular embodiment, the adder is a motion compensation circuit. 122 or the motion compensated prediction previously produced by the intra-prediction circuit 112 The block is then added to the reconstructed residual block for storage in the reference picture memory. The reconstructed video block is then used to generate a reconstructed video block. To inter-code a block in an inter-frame, a motion estimation circuit 124 and a motion compensation circuit 125 are provided. The compensation circuit 122 can use the resulting signal as a reference block.
[0028] In-loop filters are commonly used to improve coding efficiency and visual quality. For example, AVC and HEVC, as well as current versions of VVC, support non-blocking In HEVC, S filter is available to further improve the coding efficiency. An additional in-loop filter called AO (Sample Adaptive Offset) is defined. The current version of the VVC standard requires that ALF (Adaptive Feedback) be added for incorporation into the final standard. Yet another in-loop filter, called a 100-millisecond loop filter, is being actively investigated. These in-loop filter operations are optional. These operations also reduce computational complexity, helping to improve efficiency and visual quality. It can also be turned off as a decision made by the encoder 100 to avoid this. If these filter options are turned on by the encoder 100, intra prediction is usually based on unfiltered reconstructed pixels, whereas inter prediction is Note that this is based on the filtered reconstruction pixels.
[0029] FIG. 2 shows an example decoder that can use the method for bidirectional optical flow. 2 is a block diagram of a decoder 200. In an exemplary embodiment, the decoder 200 is The reconstruction-related portion of the decoder 100 is similar to that of the decoder 100. In the decoder 200, first, the incoming video bitstream 201 is decoded using entropy - Decoding by the decoding circuit 202 to derive the quantized coefficient levels and prediction relationship information The quantized coefficient levels are then converted by an inverse quantization circuit 204 and an inverse transform circuit 206. The inverse quantization circuit 204 processes the bitstream by The quantized transform coefficients are provided and decoded by the entropy decoding circuit 202. Inverse quantization, or dequantization. The inverse quantization process is carried out on each video Using the quantization parameter QPy calculated by the decoder 200 for the block , the degree of quantization to be applied, and similarly the degree of inverse quantization. The inverse transform circuit 206 performs an inverse transform, such as an inverse DCT, an inverse integer transform, or a conceptually similar Applying the inverse transform process to the transform coefficients to produce a residual block in the pixel domain .
[0030] Block prediction that can be implemented in the intra / inter mode selector circuit 208 The child mechanism includes an intra prediction circuit 210 configured to perform an intra prediction process; and / or configured to perform a motion compensation process based on the decoded prediction information. The inverse transform circuit 206 includes a motion compensation circuit 212. ... and adding the prediction residual generated by the block predictor mechanism to the prediction output generated by the block predictor mechanism. gives us a set of unfiltered reconstructed pixels. When 216 is turned on, a filtering operation is performed on these reconstructed pixels. Run it to derive the final reconstructed video for output.
[0031] When a video slice is coded as an intra-coded (I) slice, the The prediction circuit 210 predicts the signal from the previously decoded blocks of the current frame or picture. The video prediction for the current video slice is performed based on the transmitted intra prediction mode and data. It is possible to generate prediction data for an ob-block. When the image is coded as a motion-compensated (i.e., B, P, or GPB) slice, The circuit 212 receives the motion vectors and other syntax from the entropy decoding circuit 202. Produce a prediction block for the video block of the current video slice based on the factors. A prediction block is based on one of the reference pictures in one of the reference picture lists. The decoder 200 uses a default construction to generate the reference Based on the reference pictures stored in the picture memory, the reference frame list Lis t0 and List1 can be constructed.
[0032] The motion compensation circuit 212 parses the motion vectors and other syntax elements to , determine prediction information for a video block of the current video slice, and use the prediction information to The prediction block is generated for the current video block being decoded. For example, the motion compensation circuit 212 is used to encode a video block of a video slice. The received prediction mode (e.g., intra-prediction or inter-prediction) is determined. The syntax elements selected, the inter prediction slice type (for example, B slice, P slice, or is a GPB slice), one or more of the reference picture lists for the slice configuration information for each inter-coded video block of a slice, motion vectors for each inter-coded video block of the slice, the inter prediction state for each inter coded video block of the slice, and and some other information for decoding the video blocks in the current video slice. Use.
[0033] The motion compensation circuit 212 also performs the interpolation based on an interpolation filter for sub-pixel accuracy. The motion compensation circuit 212 may perform the encoding during coding of a video block. The interpolation filter used by Da100 is used to interpolate sub-integer pixels of the reference block. In an exemplary embodiment, the motion compensation circuit 21 can calculate an interpolated value for 2 determines the interpolation filter used by the encoder 100 from the received syntax elements. The motion compensation circuit 212 determines the motion vector and uses an interpolation filter to generate a prediction block. to carry out any or all of the methods described in For example, the motion compensation circuit 212 may be configured to It can be configured to perform a DOF method.
[0034] In an exemplary embodiment, the motion compensation circuit 212 calculates the first picture temporally preceding the current picture. reference picture (for example, in reference picture list 0) and a picture that is temporally further away from the current picture. For a block having a second reference picture (e.g., in reference picture list 1) It is configured to apply BDOF to the block. In response, the motion compensation circuit 212 performs the methods discussed herein (e.g., The block can be predicted according to the BDOF by applying the formula (see below).
[0035] 3A-3E show five exemplary distribution types: 4-way (FIG. 3A), 2-way (FIG. 3B), (Figure 3B), vertical bisection (Figure 3C), horizontal trisection (Figure 3D), and vertical trisection (Figure 3 E).
[0036] Referring again to FIG. 1, spatial and / or temporal prediction can be performed. Prediction (or “intra prediction”) is the prediction of an already coded signal within the same video picture or slice. The pixel values from the samples of the neighboring blocks (called reference samples) are used for the reconstruction. Spatial prediction reduces the spatial redundancy inherent in video signals. Temporal prediction (also called "inter prediction" or "motion compensated prediction") is a technique that A method for predicting a current video block using reconstructed pixels from a decoded video picture. Temporal prediction reduces the temporal redundancy inherent in a video signal. The inter prediction signal typically indicates the amount and direction of motion between the current CU and the current CU's time reference. The motion vectors are signaled by one or more motion vectors (MVs) that represent the motion of the target object. If the image corresponds to a reference picture, one reference picture index is also sent and stored in the reference picture store. It is used to identify which reference picture the temporal prediction signal comes from. and / or after temporal prediction, a mode decision block in the encoder may Based on the method, the best prediction mode is selected. Then, the prediction block is selected from the current video block. The prediction residual is then decorrelated and quantized using a transform. The residuals are inverse quantized and inverse transformed to form the reconstructed residuals, which are then retransformed back into the prediction block. In addition, it forms the reconstruction signal of the CU. For example, sample adaptive offset (SAO) and and non-blocking filters such as adaptive in-loop filters (ALFs) for further optimization. Then, the reconstructed CU can be filtered in a loop. , which are placed in the reference picture store and used to encode future video blocks. To form the output video bitstream, the coding mode (inter or inter) is selected. tra), prediction mode information, motion information, and quantized residual coefficients are all entropy-based The data is then sent to an encoding circuit 106 for further compression and packing to form a bitstream.
[0037] FIG. 4 illustrates an exemplary bidirectional optical flow (BDOF) model. As shown in FIG. Thus, the first prediction L0 in the first reference block I0 is the current block on which the prediction is being performed. The motion vector component (V x ,V y ) is identified within the first reference block I0. Similarly, a second prediction L1 in a second reference block I1 is a second time from the current block. They are located at a distance τ1. The motion vector component (-V x ,-V y The second movement A vector B is identified within the second reference block I1.
[0038] Bidirectional optical flow: Conventional bidirectional prediction in video coding is already reconstructed It is a simple combination of two temporal prediction blocks obtained from a given reference picture. Due to the constraints of block-based motion compensation, the observed difference between samples of two prediction blocks is There may be a small amount of motion remaining, thus reducing the efficiency of the motion compensated prediction. To solve this problem, VVC uses the To reduce the effects of such motion, bidirectional optical flow (BDOF) is applied. More specifically, as shown in Figure 4, BDOF is used when bi-directional prediction is used. This is a sample-by-sample motion compensation performed in addition to block-based motion compensation prediction. ×4 sub-block motion compensation (v x ,v y ) is a 6x6 window around the subblock. After BDOF is applied in Ω, the difference between the L0 predicted sample and the L1 predicted sample is minimized. More specifically, (v x ,v y ) is calculated as follows: It is derived.
[0039]
number
[0040] In this formula,
number
number
[0041] The values of S1, S2, S3, S5, and S6 are calculated as follows:
number
[0042] In this formula, Ω is a set of adjacent predictor subblocks around one subblock of prediction L0 and L1. is a sample and is given by the following formula:
number
[0043] In this formula, I (k) (i,j) is generated with moderately high precision (i.e., 16 bits) The predicted sample at sample position (i,j) of the predicted signal in the list k, k=0,1 It is
number
number
[0044] Based on the motion correction derived in Eq. (1), the motion based on the optical flow model is The final bidirectional prediction of the CU is calculated by interpolating the L0 prediction and the L1 prediction along the trajectory. The measurement sample is calculated as shown in the following equation (5).
number
[0045] In this formula, shift and ο offset For bi-directional prediction, the L0 prediction signal and and a right shift value and an offset value applied to combine the L1 prediction signal, They are equal to 15-BD and 1<<(14-BD)+2·(1<<13), respectively. Table 1 shows the specific bit widths of the intermediate parameters involved in the BDOF process. As shown in Figure 1, the internal bit width of the entire BDOF process does not exceed 32 bits. In equation (1), the multiplication by the worst possible input is v x S 2,m The product of The output bit width is S 2,m and v x 15 bits and 4 bits respectively. Therefore, a 15-bit multiplier is sufficient for BDOF.
[0046] [Table 1] [Table 2]
[0047] BDOF can enhance the efficiency of bidirectional prediction, but further improvements in the design of BDOF are needed. More specifically, there is an opportunity to improve the VVC for controlling the bit width of intermediate parameters. The following problems with existing BDOF designs in are identified in this disclosure:
[0048] First, as shown in Table 1, the parameters θ(i,j) (i.e., L0 prediction and L 1 prediction), as well as the parameter ψ x (i,j) and ψy (i,j) (i.e. The sum of the horizontal and vertical gradient values for L0 and L1) is the same bit size of 11 bits. Such an approach allows for a global constraint on the internal bit width relative to the BDOF. This makes it easier to control, but this method is not optimal in terms of the accuracy of the derived motion compensation. This is because, as shown in equation (4), these gradient values are The high-pass nature of such a process makes it possible to The gradients obtained are filtered out to eliminate noise, e.g., noise captured in the original video and noise during the encoding process. This is because the method is not reliable in the presence of coding noise generated in the high bit This means that it is not always useful to represent gradient values by width.
[0049] In addition, as shown in Table 1, the maximum bit width usage of the entire BDOF process is Motion Compensation y First, S6 (27 bits) is shifted left by 3 bits. Then, ((v x S 2,m )<<12+v x S 2,s ) / 2(30 bits) is subtracted Therefore, the maximum bit width of the current design is equal to 31 bits. Hardware implementations typically use encoding processes with a maximum internal bit width greater than 16 bits. However, the existing design is implemented in a 32-bit implementation. This does not fully utilize the available dynamic range of the BDOF derived motion. This can result in unnecessary loss of accuracy in the correction.
[0050] In an exemplary embodiment, such as the exemplary method illustrated in FIG. 5, an exemplary bit width control method includes: The above problems of the bit-width control method for the conventional BDOF design are addressed.
[0051] First, to overcome the negative impact of gradient estimation error, in an exemplary embodiment, Equation (4) The next gradient value at
number
number
number
number
[0052] In addition, the entire BDOF process is optimized to operate with an appropriate internal bit width. To control the body, the variable ψ x (i,j), ψ y Calculation of (i,j) and θ(i,j) An additional bit shift of n adj is introduced as follows:
number
[0053] Table 2 below shows each intermediate bit width when the exemplary bit width control method is applied to the BDOF. Table 2 shows the corresponding bit width of the parameters. The changes applied in the exemplary bit width control method are shown in Table 2. The exemplary bit width control method allows for the internal binarization of the entire BDOF process. In addition, this exemplary method ensures that the maximum bit width is It is now just 32-bit, which means that the available dynamic Moreover, as shown in Table 2, The worst case input multiplication is v x S 2,m The input S 2,m is 14 bits , input v x is 6 bits. Therefore, similar to the conventional BDOF method, the exemplary method When this applies, a single 16-bit multiplier is still large enough.
[0054] Referring to Table 2, the right-shifted bits applied in equations (6) and (7) are The modification to the number of parameters, compared to the traditional BDOF method shown in Table 1, ψ x (i,j), ψ y (i,j) and θ(i,j) have different dynamic ranges. The three parameters are represented in the same dynamic range (i.e., 21 bits). Such a change increases the bit width of the internal parameters S1, S2, S3, S5, and S6. Therefore, in some cases, the maximum bit width of the internal BDOF process can be increased to 32. Therefore, to ensure a 32-bit implementation, To achieve this, two additional clipping operations are introduced when calculating the S2 and S6 values. More specifically, in the exemplary method, the values of two parameters are calculated as follows: It is calculated.
number
[0055] In this formula, B2 and B6 control the output dynamic range of S2 and S6, respectively. Unlike the gradient calculation, the clipping operation in (8) is It is applied once to calculate the motion compensation for each 4 × 4 subblock in a CU of one BDOF. Note that the 4×4 unit is used for the call. The corresponding increase in complexity due to the clipping operations introduced in the method is very small. .
[0056] In an exemplary embodiment, the final bi-directional prediction of a CU is the first prediction L0 and the second prediction L1. Measurement L1, motion compensation (v x ,v y ), as well as the first gradient value
number
number
[0057] [Table 3] [Table 4]
[0058] FIG. 5 illustrates an example method 30 for predicting a block of video data according to an example embodiment. 5 is a flowchart for the method shown in FIG. and / or will be described with respect to the decoder 200 of FIG. 2. However, in another example, Other devices may also be configured to carry out the methods.
[0059] In an exemplary embodiment, the encoder 100 first The picture being coded is then decoded for use as a reference picture. These steps are not shown in the exemplary method of FIG. receives a block of video data. The block may be, for example, a current The prediction unit (PU) of the coding unit (CU) of a picture (or a slice of a picture) The encoder 100 may perform various prediction methods, such as intra prediction or inter prediction. Determine which of the modes to use to predict a block, and When it is determined, whether to use unidirectional inter prediction or bidirectional inter prediction. In the exemplary method of FIG. 5, the encoder 100 , determine to predict the block using bi-directional prediction. For example, the encoder 100 may , various possible prediction modes (e.g., one or more intra-prediction modes, and reference unidirectional or from various decoded reference pictures stored in the reference picture memory; Calculate the rate-distortion metric for the bi-prediction block and It may be determined that the best rate-distortion metric is achieved using the eigenvalues.
[0060] The encoder 100, for example the motion estimation circuit 124, performs bi-directional prediction to estimate the block size. A motion vector for the current block can be further calculated. The first reference picture to be displayed before the current picture and the second reference picture to be displayed after the current picture. For example, the first reference picture may point to a second reference picture that is to be used as the first reference picture. A second reference picture may occur in reference picture list 0 (RefPicList0). The chunk can occur in reference picture list 1 (RefPicList1).
[0061] In one embodiment, the encoder 100 calculates a first motion vector (MV0) at a first temporal distance. POC value for the current picture and POC for the first reference picture at a distance τ0 A first POC difference between the value and a second motion vector (MV1) is referenced at a second time distance τ1. between the POC value for the current picture and the POC value for the second reference picture The second POC difference is determined. MV0 and MV1 are the x and y components of the For example, MV0 can include MV x0 and M.V. y0 Can contain ingredients , here MV x0 to determine a position to identify a reference block in the first reference picture. indicates the horizontal offset relative to the location of the block in the current picture, as in MV y0 teeth, Similarly, MV1 indicates the vertical offset of MV x1 and M.V. y1 May contain ingredients Here, MV x1 determines a location to identify a reference block in the second reference picture indicates the horizontal offset relative to the location of the block in the current picture, as in MV y1 teeth , indicates the vertical offset.
[0062] In an exemplary embodiment, the first reference picture and the second reference picture are decoded ( 302). The second reference picture is different from the first reference picture and is The first reference picture is displayed before the current picture, and the second reference picture is displayed after the current picture. In an exemplary embodiment, the current block of the current picture is compared to the first reference picture. A first motion vector (MV0) to a first prediction L0 of the current block is determined (304). A second motion vector (MV1) is determined from the first block to a second prediction L1 of the second reference picture. In a particular exemplary embodiment, the first reference picture is determined from the current picture (306). The first reference point from the current block of the current picture at the first time distance (τ0) to the A first motion vector (MV0) to a first prediction L0 of the reference picture is determined, and a first motion vector (MV1) to a first prediction L1 of the reference picture is determined. The second reference picture is located at a second time distance (τ1) from the current block to the second reference picture. A second motion vector (MV1) to a second prediction of the second reference picture L1 is determined.
[0063] The encoder 100 then uses these values to generate a first prediction L0 and a second prediction L 1, the motion compensation (v x ,v y ) In an exemplary embodiment, the following first gradient for the first prediction L0 is calculated (308): value
number
number
[0064] In an exemplary embodiment, (v x ,v y ) is calculated using equation (1) and equation (7 ), additional bits are used to calculate the values of S1, S2, S3, S5, and S6. Shift n adj However, there are multiple variables ψ x (i,j), ψ y (i,j), and θ(i,j For example, the values of S2 and S6 are calculated using equation (8). In an exemplary embodiment, the S2 and S6 values are calculated based on the first clip, as shown in equation (8). is calculated by introducing a ping operation and a second clipping operation, where B 2 is a parameter for controlling the output dynamic range of S2, and B6 is a parameter for controlling the output dynamic range of S6 The parameter for controlling the output dynamic range of the first prediction L0 and the second prediction L1, motion compensation (v x , v y ), the first gradient value
number
number
[0065] The exemplary method illustrated in FIG. 5 represents one example of a method for encoding video data. a step of decoding the first reference picture and the second reference picture, The reference picture is different from the first reference picture and the first reference picture is before the current picture. and the second reference picture is displayed after the current picture, A first motion from a current block of the first picture to a first reference block of a first reference picture. determining a vector (MV0) from the current block to the second reference picture; determining a second motion vector (MV1) to a second reference block; and coding the current block according to the head-dot-field (BDOF). .
[0066] In one or more examples, the functions described may be implemented in hardware, software, firmware, or other If implemented in software, In some cases, these functions may be represented as one or more instructions or code on a computer-readable medium. The present invention can be implemented in a variety of ways, including by storing the program in a computer-readable medium or by transmitting the program via a computer-readable medium, and can be implemented in a variety of ways, including by performing the program on a hardware-based system. The computer-readable medium may be a data storage medium. A computer-readable storage medium corresponding to a tangible medium such as a Any program that facilitates the transfer of a computer program from one place to another in accordance with In this manner, the computer-readable medium may include a communication medium, including a (1) a non-transitory tangible computer-readable storage medium, or (2) a signal or carrier The data storage medium may correspond to a communication medium such as a transmission of waves. To extract instructions, code, and / or data structures for implementing the Examples, Accessed by one or more computers or one or more processors The computer program product may be any available medium capable of carrying out the It may include a computer readable medium.
[0067] Additionally, the above methods may be implemented using an apparatus that includes one or more circuits. Such circuits include application specific integrated circuits (ASICs), digital signal processors, (DSP), Digital Signal Processing Devices (DSPD), Programmable Logic Devices (P LD), Field Programmable Gate Array (FPGA), Controller, Micro A controller, microprocessor, or other electronic components may be included in the device. In combination with other hardware or software components to carry out the methods described above. These circuits can be used in combination with each other. A module, unit, or subunit is a module that at least partially implements one or more circuits. It can be implemented using.
[0068] Other embodiments of the present invention will become apparent from consideration of the present specification and practice of the present invention disclosed above. This application is based on the general principles of the invention and is not intended to be limiting or limiting unless otherwise specified. Any modifications, uses, or variations of the invention, including departures from the present disclosure that come within the scope of customary practice, are expressly intended to be included herein. It is intended that the present specification and examples be considered as exemplary only. The true scope and spirit of the invention is indicated by the following claims. It is intended that
[0069] The present invention is not limited to the precise examples described above and illustrated in the accompanying drawings. Various modifications and alterations may be made without departing from the scope of the present invention. It will be understood that the scope of the present invention is to be limited only by the appended claims. As intended.
Claims
1. One or more processors; and a method executed by the one or more processors. The present invention is implemented on a computing device having a memory for storing a plurality of programs. It controls the bit width for bidirectional optical flow (BDOF) for video coding. A method of controlling A step of decoding a first reference picture and a second reference picture, the second reference picture being is different from the first reference picture, and the first reference picture is different from the current the first reference picture is displayed before the current picture and the second reference picture is displayed after the current picture. Steps A first prediction L of the first reference picture from the current block of the current picture 0 fart The first motion vector (MV 0 ) A second prediction L of the second reference picture from the current block 1 The second movement vector to Tor (MV 1 ) The first prediction L 0 and the second prediction L 1 By minimizing the difference between Motion compensation of the CU (v x , v y ) and The first prediction L 0 First gradient value for [0010] And the second prediction L 1 The second gradient value for [0025] and calculating The first prediction L 0 and the second prediction L 1 , the motion correction (v x , v y ), Gradient value of 1 [0030] and the second gradient value [0045] and calculating a final bi-prediction of the CU based on
2. (v x , v y 2. The method of claim 1 , wherein the value of ) is calculated as follows: [0050]
3. S 1 , S 2 , S 3 , S 5 , and S 6 The value of [006] where Ω is the first prediction L 0 and the second prediction L 1 One sub-sub 3. The method of claim 2, wherein the set of adjacent predicted samples around a lock.
4. n c The method of claim 2 , wherein the value of is set equal to two.
5. th BDOF The method of claim 2 , wherein the value of is set equal to two.
6. Multiple variables ψ x (i, j), ψ y (i,j), and θ(i,j) are calculated using additional bits. Shift n adj The method further comprises the step of introducing [0070] The method according to claim 2, wherein
7. n adj The method of claim 6 , wherein the value of is set equal to two.
8. By introducing the first clipping operation and the second clipping operation, S 2 and S 6 wherein the first clipping operation further comprises calculating a value of [0080] It is calculated as follows: The second clipping operation comprises: [0090] It is calculated as follows: B 2 But, S 2 is a parameter that controls the output dynamic range of 6 but, S 6 7. The method according to claim 6, wherein the parameter is for controlling the output dynamic range of the Law.
9. Gradient Value [0089] and the gradient value ##EQU00011## The step of calculating a first predicted position of the first prediction and a second predicted position of the second prediction is performed.
2. The method of claim 1 further comprising the step of calculating horizontal and vertical gradients at each of the predicted locations. The method described.
10. the horizontal gradient at each of the first predicted location and the second predicted location is ##EQU00012## It is calculated as follows: the vertical gradient at each of the first predicted location and the second predicted location is ##EQU00013## The method of claim 9, wherein the eigenvalue is calculated as follows:
11. one or more processors; a non-transitory storage device coupled to the one or more processors; a plurality of programs stored in the non-transitory storage device; When the plurality of programs are executed by the one or more processors, A step of decoding a first reference picture and a second reference picture, the second reference picture being is different from the first reference picture, and the first reference picture is different from the current the first reference picture is displayed before the current picture and the second reference picture is displayed after the current picture. Steps A first prediction L of the first reference picture from the current block of the current picture 0 fart The first motion vector (MV 0 ) A second prediction L of the second reference picture from the current block 1 The second movement vector to Tor (MV 1 ) The first prediction L 0 and the second prediction L 1 By minimizing the difference between Motion compensation of the CU (v x , v y ) and The first prediction L 0 The first gradient value for ##EQU00014## And the second prediction L 1 The second gradient value for ##EQU00015## and calculating The first prediction L 0 and the second prediction L 1 , the motion correction (v x , v y ), Gradient value of 1 ##EQU00016## and the second gradient value ##EQU00017## and calculating a final bi-prediction of the CU based on A computing device that causes a computing device to execute a program.
12. (v x , v y 12. The computer program of claim 11, wherein the value of Device. [0018]
13. S 1 , S 2 , S 3 , S 5 , and S 6 The value of [0019] where Ω is the first prediction L 0 and the second prediction L 1 One sub-sub 13. The computing device of claim 12, wherein the set of adjacent predicted samples around a lock. device.
14. n c 13. The computing device of claim 12, wherein the value of is set equal to 2. 。
15. th BDOF 13. The computing device of claim 12, wherein the value of is set equal to 2. device.
16. Multiple variables ψ x (i, j), ψ y (i,j), and θ(i,j) are calculated using additional bits. Shift n adj The method further comprises the step of introducing [0020] 13. The computing device of claim 12, wherein:
17. n adj 17. The computing device of claim 16, wherein the value of is set equal to 2. Chair.
18. By introducing the first clipping operation and the second clipping operation, S 2 and S 6 wherein the first clipping operation further comprises calculating a value of ##EQU00021## It is calculated as follows: The second clipping operation comprises: [0022] It is calculated as follows: B 2 But, S 2 is a parameter that controls the output dynamic range of 6 but, S 6 17. The parameter according to claim 16, for controlling the output dynamic range of Computing device.
19. Gradient Value [0023] and the gradient value ##EQU00024## The step of calculating a first predicted position of the first prediction and a second predicted position of the second prediction is performed.
12. The method of claim 11, further comprising the step of calculating horizontal and vertical gradients at each of the predicted locations. A computing device as described in
20. the horizontal gradient at each of the first predicted location and the second predicted location is [0025] It is calculated as follows: the vertical gradient at each of the first predicted location and the second predicted location is [0026] The computing device of claim 13 , wherein:
21. For execution by a computing device having one or more processors A non-transitory computer-readable storage medium storing a plurality of programs, When executed by the one or more processors, A step of decoding a first reference picture and a second reference picture, the second reference picture being is different from the first reference picture, and the first reference picture is different from the current the first reference picture is displayed before the current picture and the second reference picture is displayed after the current picture. Steps A first prediction L of the first reference picture from the current block of the current picture 0 fart The first motion vector (MV 0 ) A second prediction L of the second reference picture from the current block 1 The second movement vector to Tor (MV 1 ) The first prediction L 0 and the second prediction L 1 By minimizing the difference between Motion compensation of the CU (v x , v y ) and The first prediction L 0 The first gradient value for [0027] And the second prediction L 1 The second gradient value for [0028] and calculating The first prediction L 0 and the second prediction L 1 , the motion correction (v x , v y ), Gradient value of 1 [0029] and the second gradient value [0030] and calculating a final bi-prediction of the CU based on A non-transitory computer-readable storage medium for execution on a computing device.
22. (v x , v y 22. The non-transient computing device of claim 21, wherein the value of A computer-readable storage medium. [0031]
23. S 1 , S 2 , S 3 , S 5 , and S 6 The value of [0032] where Ω is the first prediction L 0 and the second prediction L 1 One sub-sub 23. The non-transitory computer of claim 22, wherein the set of adjacent predicted samples around a lock. A data-readable storage medium.
24. Multiple variables ψ x (i, j), ψ y (i,j), and θ(i,j) are calculated using additional bits. Shift n adj The method further comprises the step of introducing [Equation 33] 23. The non-transitory computer-readable storage medium of claim 22, wherein:
25. By introducing the first clipping operation and the second clipping operation, S 2 and S 6 and further comprising the step of calculating the value of The first clipping operation comprises: [0034] It is calculated as follows: The second clipping operation comprises: [Equation 35] It is calculated as follows: B 2 But, S 2 is a parameter that controls the output dynamic range of 6 but, S 6 25. The parameter according to claim 24, for controlling the output dynamic range of A non-transitory computer-readable storage medium.
26. Gradient Value [0036] and the gradient value [Equation 37] The step of calculating a first predicted position of the first prediction and a second predicted position of the second prediction is performed.
22. The method of claim 21 further comprising the step of calculating horizontal and vertical gradients at each of the predicted locations. A non-transitory computer-readable storage medium as described above.
27. the horizontal gradient at each of the first predicted location and the second predicted location is [Equation 38] It is calculated as follows: the vertical gradient at each of the first predicted location and the second predicted location is [0039] 27. The non-transitory computer-readable storage medium of claim 26, wherein the non-transitory computer-readable storage medium is calculated as follows:
Citation Information
Patent Citations
Bitwidth Control for Bidirectional Optical Flow
JP7474262B2
Image encoding method, image decoding method, image encoding device, image decoding device and program
WO2018123317A1
Cited By
Semiconductor device
CN121712057A
