Method and device for video processing and medium

Abstract

Embodiments of the present disclosure provide a solution for video processing. A method for video processing is presented. The method includes iteratively applying multiple rounds of bidirectional optical flow (BDOF) processes to a current video block of a video for a conversion between the current video block and a bitstream of the video, the multiple rounds of BDOF processes being of the same type; and performing conversion based on a result of applying the multiple rounds of BDOF processes.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to video processing techniques, and more specifically, to bidirectional optical flow (BDOF) processes. Background Technology

[0002] Today, digital video capabilities are being applied to all aspects of people's lives. Various video compression technologies have been proposed for video encoding / decoding, such as MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-TH.265 High Efficiency Video Codec (HEVC) standard, and Multifunctional Video Codec (VVC) standard. However, the overall expectation is to further improve the encoding and decoding quality of video encoding and decoding technologies. Summary of the Invention

[0003] Embodiments of this disclosure provide a solution for video processing.

[0004] In a first aspect, a method for video processing is proposed. This method includes: iteratively applying a multi-round bidirectional optical flow (BDOF) process of the same type to the current video block for conversion between the current video block and the video bitstream; and performing the conversion based on the result of applying the multi-round BDOF process.

[0005] Based on the method according to the first aspect of this disclosure, the same type of BDOF process can be performed iteratively for the current video block. Compared with conventional solutions where the same type of BDOF process is not performed iteratively, the proposed method can advantageously refine the sample values ​​or MV of the current video block more effectively. Thus, the encoding and decoding quality can be improved.

[0006] In a second aspect, an apparatus for video processing is provided. The apparatus includes a processor and a non-transitory memory having instructions thereon. When executed by the processor, the instructions cause the processor to perform the method according to the first aspect of this disclosure.

[0007] In a third aspect, a non-transitory computer-readable storage medium is proposed. This non-transitory computer-readable storage medium stores instructions that cause a processor to execute the method according to the first aspect of this disclosure.

[0008] In a fourth aspect, another non-transitory computer-readable recording medium is proposed. This non-transitory computer-readable recording medium stores a bitstream of video, which is generated by a method performed by a device for video processing. The method includes: iteratively applying a multi-round bidirectional optical flow (BDOF) process to a current video block, wherein the multi-round BDOF processes are of the same type; and generating a bitstream based on the result of applying the multi-round BDOF process.

[0009] In the fifth aspect, a method for storing a bitstream of video is proposed. This method includes: iteratively applying a multi-round bidirectional optical flow (BDOF) process to the current video block, wherein the multi-round BDOF processes are of the same type; generating a bitstream based on the results of applying the multi-round BDOF process; and storing the bitstream in a non-transitory computer-readable recording medium.

[0010] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0011] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0012] Figure 1 A block diagram illustrating an example video codec system according to some embodiments of the present disclosure is shown; Figure 2 A block diagram illustrating a first example video encoder according to some embodiments of the present disclosure is shown; Figure 3 A block diagram illustrating an example video decoder according to some embodiments of the present disclosure is shown; Figure 4 The Extended Codec Unit (CU) region used in BDOF is shown; Figure 5 This shows the refinement of motion vectors on the decoding side; Figure 6 The diamond-shaped area in the search region is shown; Figure 7 The weights generated using the example Gaussian distribution are shown; Figure 8 The weights generated using another example Gaussian distribution are shown; Figure 9 The weights generated using yet another example of a Gaussian distribution are shown; Figure 10The weights generated using yet another example of a Gaussian distribution are shown; Figure 11 Different filter shapes are shown for application to the data; Figure 12 A flowchart of a method for video processing according to embodiments of the present disclosure is shown; and Figure 13 A block diagram of a computing device in which various embodiments of the present disclosure may be implemented is shown.

[0013] Throughout all the accompanying figures, the same or similar reference numerals generally refer to the same or similar elements. Detailed Implementation

[0014] The principles of this disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art understand and implement this disclosure, and do not imply any limitation on the scope of this disclosure. In addition to the methods described below, the disclosure described herein can be implemented in various other ways.

[0015] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0016] The terms "an embodiment," "embodiment," "example embodiment," etc., used in this disclosure refer to embodiments that may include specific features, structures, or characteristics, but not every embodiment is required to include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an example embodiment, it is claimed that, whether explicitly described or not, such a feature, structure, or characteristic affecting its relation to other embodiments is within the knowledge of those skilled in the art.

[0017] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “having,” “containing,” and / or “comprising” as used herein indicate the presence of the said features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0019] Example Environment Figure 1 This is a block diagram illustrating an example video encoding / decoding system 100 from which the techniques of this disclosure may be utilized. As shown, the video encoding / decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0020] Video source 112 may include sources such as video capture devices. Examples of video capture devices include, but are not limited to, interfaces for receiving video data from video content providers, computer graphics systems for generating video data, and / or combinations thereof.

[0021] Video data may include one or more images. Video encoder 114 encodes the video data from video source 112 to generate a bitstream. The bitstream may include a sequence of bits forming an encoded representation of the video data. The bitstream may include encoded images and associated data. An encoded image is an encoded representation of an image. Associated data may include sequence parameter sets, image parameter sets, and other syntax structures. I / O interface 116 may include a modulator / demodulator and / or a transmitter. Encoded video data can be directly transmitted to destination device 120 via network 130A through I / O interface 116. Encoded video data may also be stored on storage medium / server 130B for access by destination device 120.

[0022] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may acquire encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to a user. The display device 122 may be integrated with the destination device 120, or it may be external to the destination device 120, which is configured to interface with an external display device.

[0023] The video encoder 114 and the video decoder 124 can operate according to video compression standards such as the High Efficiency Video Codec (HEVC) standard, the Multi-Functional Video Codec (VVC) standard, and other existing and / or future standards.

[0024] Figure 2 This is a block diagram illustrating an example of a video encoder 200 according to some embodiments of the present disclosure. The video encoder 200 may be... Figure 1 An example of a video encoder 114 in system 100 is shown.

[0025] The video encoder 200 can be configured to implement any or all of the technologies disclosed herein. Figure 2 In the example, the video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 200. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0026] In some embodiments, the video encoder 200 may include a segmentation unit 201, a prediction unit 202, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214. The prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra-frame prediction unit 206.

[0027] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit can perform prediction in an IBC mode, in which at least one reference picture is the picture in which the current video block is located.

[0028] Furthermore, although some components (such as motion estimation unit 204 and motion compensation unit 205) can be integrated, for interpretable purposes, these components are... Figure 2The examples are shown separately.

[0029] The segmentation unit 201 can segment an image into one or more video blocks. The video encoder 200 and the video decoder 300 can support various video block sizes.

[0030] The mode selection unit 203 can, for example, select one of several coding modes (intra-coding or inter-coding) based on the error result, and provide the resulting intra-coded or inter-coded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coded block for use as a reference image. In some examples, the mode selection unit 203 can select an intra-inter-prediction joint prediction (CIIP) mode, in which prediction is based on inter-prediction signals and intra-prediction signals. In the case of inter-prediction, the mode selection unit 203 can also select a resolution for the block based on the motion vector (e.g., sub-pixel precision or integer pixel precision).

[0031] To perform inter-frame prediction on the current video block, motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from buffer 213 with the current video block. Motion compensation unit 205 can determine the predicted video block for the current video block based on the motion information and decoded samples of images from buffer 213 other than the image associated with the current video block.

[0032] The motion estimation unit 204 and the motion compensation unit 205 can perform different operations on the current video block, for example, depending on whether the current video block is in an I-strip, P-strip, or B-strip. As used herein, an "I-strip" can refer to a portion of an image composed of macroblocks, all of which are based on macroblocks within the same image. Furthermore, as used herein, in some aspects, "P-strip" and "B-strip" can refer to portions of an image composed of macroblocks independent of macroblocks within the same image.

[0033] In some examples, motion estimation unit 204 can perform unidirectional prediction on the current video block, and can search reference images in list 0 or list 1 to find a reference video block for the current video block. Motion estimation unit 204 can then generate a reference index indicating the reference image containing the reference video block in list 0 or list 1, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 204 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0034] Alternatively, in other examples, motion estimation unit 204 can perform bidirectional prediction on the current video block. Motion estimation unit 204 can search reference images in list 0 to find a reference video block for the current video block, and can also search reference images in list 1 to find another reference video block for the current video block. Motion estimation unit 204 can then generate multiple reference indices and multiple motion vectors, the multiple reference indices indicating multiple reference images containing multiple reference video blocks in lists 0 and 1, and the multiple motion vectors indicating multiple spatial displacements between the multiple reference video blocks and the current video block. Motion estimation unit 204 can output the multiple reference indices and multiple motion vectors of the current video block as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current video block based on the multiple reference video blocks indicated by the motion information of the current video block.

[0035] In some examples, the motion estimation unit 204 can output a complete set of motion information for use in the decoder's decoding process. Alternatively, in some embodiments, the motion estimation unit 204 can transmit the motion information of the current video block via a signal, referencing the motion information of another video block. For example, the motion estimation unit 204 can determine that the motion information of the current video block is sufficiently similar to the motion information of neighboring video blocks.

[0036] In one example, the motion estimation unit 204 may indicate a value in the syntax structure associated with the current video block that indicates to the video decoder 300 that the current video block has the same motion information as another video block.

[0037] In another example, motion estimation unit 204 can identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 300 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0038] As discussed above, the video encoder 200 can transmit motion vectors via signals in a predictive manner. Two examples of predictive signaling techniques that can be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and Merge Pattern Signaling.

[0039] Intra-prediction unit 206 can perform intra-prediction on the current video block. When intra-prediction unit 206 performs intra-prediction on the current video block, it can generate prediction data for the current video block based on decoded samples from other video blocks in the same frame. The prediction data for the current video block can include the predicted video block and various syntax elements.

[0040] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) multiple predicted video blocks from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of the samples in the current video block.

[0041] In other examples, such as in skip mode, residual data for the current video block may not exist, and residual generation unit 207 may not perform a subtraction operation.

[0042] The transform processing unit 208 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0043] After the transform processing unit 208 generates a transform coefficient video block associated with the current video block, the quantization unit 209 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0044] The inverse quantization unit 210 and the inverse transform unit 211 can apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 212 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 202 to produce a reconstructed video block associated with the current video block for storage in the buffer 213.

[0045] After the video block is reconstructed by reconstruction unit 212, a loop filtering operation can be performed to reduce video block artifacts in the video block.

[0046] Entropy encoding unit 214 can receive data from other functional components of video encoder 200. When entropy encoding unit 214 receives data, it can perform one or more entropy encoding operations to generate entropy-encoded data and output a bitstream including the entropy-encoded data.

[0047] Figure 3 This is a block diagram illustrating an example of a video decoder 300 according to some embodiments of the present disclosure. The video decoder 300 may be... Figure 1 An example of video decoder 124 in system 100 is shown.

[0048] The video decoder 300 can be configured to perform any or all of the technologies disclosed herein. Figure 3In the example, the video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 300. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0049] exist Figure 3 In the example, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-frame prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally contrasted with the encoding process described with respect to the video encoder 200.

[0050] Entropy decoding unit 301 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded blocks of video data). Entropy decoding unit 301 can decode the entropy-encoded video data, and motion compensation unit 302 can determine motion information from the entropy-decoded video data, including motion vectors, motion vector precision, reference picture list indices, and other motion information. Motion compensation unit 302 can determine such information, for example, by performing AMVP and Merge mode. AMVP is used, which involves deriving several most likely candidates based on data from neighboring PBs and reference pictures. Motion information typically includes horizontal motion vector displacement values ​​and vertical motion vector displacement values, one or two reference picture indices, and, in the case of a prediction region in a B-strip, an identifier of which reference picture list is associated with each index. As used herein, in some aspects, "Merge mode" may refer to deriving motion information from spatially or temporally neighboring blocks.

[0051] The motion compensation unit 302 can generate motion compensation blocks, possibly by performing interpolation based on an interpolation filter. Identifiers for interpolation filters used with sub-pixel precision can be included in the syntax elements.

[0052] The motion compensation unit 302 can use the interpolation filter used by the video encoder 200 during the encoding of the video block to calculate the interpolated values ​​of sub-integer pixels for the reference block. The motion compensation unit 302 can determine the interpolation filter used by the video encoder 200 based on the received syntax information, and the motion compensation unit 302 can use the interpolation filter to generate the prediction block.

[0053] Motion compensation unit 302 may use at least some of the syntax information to determine the size of the blocks used to encode the encoded video sequence (multiple frames) and / or (multiple stripes), segmentation information describing how each macroblock of the image of the encoded video sequence is segmented, a pattern indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a “strip” can refer to a data structure that can be decoded independently of other stripes of the same image in terms of entropy encoding / decoding, signal prediction, and residual signal reconstruction. A strip can be an entire image or a region of an image.

[0054] Intra-prediction unit 303 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Dequantization unit 304 dequantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 301. Inverse transform unit 305 applies an inverse transform.

[0055] The reconstruction unit 306 can obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 302 or the intra-frame prediction unit 303. If necessary, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 307, which provides a reference block for subsequent motion compensation / intra-frame prediction and also generates decoded video for presentation on a display device.

[0056] Some exemplary embodiments of this disclosure will be described in detail below. It should be noted that section headings are used in this document for ease of understanding and not to limit the embodiments disclosed in a section to that section only. Furthermore, while some embodiments are described with reference to multi-functional video codecs or other specific video codecs, the disclosed techniques are also applicable to other video codec techniques. Additionally, although some embodiments describe video encoding steps in detail, it should be understood that the corresponding decoding steps for decoding will be implemented by the decoder. Furthermore, the term video processing includes video encoding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another or at different compression bitrates.

[0057] 1. Brief Overview This disclosure relates to video / image coding and decoding technologies. Specifically, it relates to bidirectional optical flow. It can be applied to existing video coding and decoding standards such as HEVC and VVC, or to next-generation video coding and decoding standards such as ECM that go beyond VVC. It can also be applied to future video coding and decoding standards or video codecs.

[0058] 2. Introduction Video codec standards have primarily evolved through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed the H.261 and H.263 standards, while ISO / IEC developed MPEG-1 and MPEG-4 Vision. The two organizations jointly developed the H.262 / MPEG-2 video standard, the H.264 / MPEG-4 Advanced Video Coding (AVC) standard, and the H.265 / HEVC standard. Starting with H.262, video codec standards are based on a hybrid video codec architecture, utilizing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was established in 2015 by VCEG and MPEG. As of July 2020, it had also completed the Multi-Functional Video Coding (VVC) standard, aiming to further reduce the bitrate by 50% and provide a range of additional features. Following the completion of VVC, activities beyond VVC have begun. The description of the additional tools on top of the VVC tools has been summarized in “Algorithmic Description of Enhanced Compression Model 5 (ECM 5)” by M. Coban, F. Léannec, K. Naser and J. Ström (26th JVET Conference, April 20-29, 2022: via teleconference, document JVET-Z2025), and its reference software is named ECM.

[0059] 2.1 Bidirectional Optical Flow (BDOF) in VVC The Bidirectional Optical Flow (BDOF) tool is included in VVC. BDOF (formerly known as BIO) was included in JEM. Compared to the JEM version, the BDOF in VVC is a simpler version that requires far fewer computations, especially in terms of the number of multiplications and the size of the multipliers.

[0060] BDOF is used to refine the bidirectional prediction signal of the CU at the 4×4 sub-block level. BDOF is applied to the CU if all of the following conditions are met: - CU is encoded and decoded using a "true" bidirectional prediction mode, that is, one of the two reference images is displayed before the current image in the order of display, and the other is displayed after the current image in the order of display.

[0061] - The distances from the two reference images to the current image (i.e., the difference in POC) are the same.

[0062] - Both reference images are short-term reference images.

[0063] - CU is not encoded or decoded using affine mode or SbTMVP Merge mode.

[0064] - The CU has more than 64 luminance samples.

[0065] - Both the CU height and CU width are greater than or equal to 8 luminance samples.

[0066] - The BCW weight index indicates equal weights.

[0067] - WP is for cases where the current CU is not enabled.

[0068] - CIIP mode is not used in the current CU.

[0069] BDOF is applied only to the luminance component. As its name suggests, the BDOF mode is based on the concept of optical flow, which assumes that the motion of the object is smooth. For each 4×4 sub-block, motion refinement is calculated by minimizing the difference between the L0 and L1 predicted samples. Motion refinement is then used to adjust the bidirectional prediction sample values ​​in the 4x4 sub-blocks. The following steps are applied during the BDOF process.

[0070] First, the horizontal and vertical gradients of the two predicted signals are calculated by directly calculating the difference between two neighboring sample points. , , ,Right now,

[0071] in It is a list k ( k The coordinates of the predicted signal in (=0,1) The sample value at the location, and shift1 is calculated based on the luminance bit depth bitDepth as shift1 = max(6, bitDepth-6).

[0072] Then, the autocorrelation and cross-correlation of the gradients. , , , and Calculated as:

[0073] in

[0074] in It is a 6x6 window surrounding a 4x4 sub-block, and and The values ​​were set to min(1, bitDepth - 11) and min(4, bitDepth - 8) respectively.

[0075] Motion refinement Then, the following formula is derived using cross-correlation and autocorrelation terms:

[0076] in . It is a floor function, and .

[0077] Based on motion refinement and gradients, the following adjustments are calculated for each sample point in the 4×4 sub-block:

[0078] Finally, the BDOF samples of CU are calculated by adjusting the bidirectional prediction samples as follows:

[0079] These values ​​were chosen so that the multiplier in the BDOF process does not exceed 15 bits, and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits.

[0080] To derive the gradient values, a list of values ​​outside the current CU boundary needs to be generated. Some predicted samples .like Figure 4 As shown, BDOF in VVC uses an extended row / column around the CU boundary. To control the computational complexity of generating prediction samples outside the boundary, prediction samples in the extended region (white area) are generated by directly taking reference samples at nearby integer positions (using the floor() operation on the coordinates) without interpolation, and a normal 8-tap motion-compensated interpolation filter is used to generate prediction samples inside the CU (gray area). These extended sample values ​​are used only for gradient calculation. For the remaining steps in the BDOF process, if any samples and gradient values ​​outside the CU boundary are needed, they are filled from their nearest neighbors (i.e., repeated).

[0081] When the width and / or height of a CU is greater than 16 luminance samples, it will be divided into sub-blocks with a width and / or height equal to 16 luminance samples, and the sub-block boundaries will be considered as CU boundaries in the BDOF process. The maximum cell size for the BDOF process is limited to 16x16. The BDOF process can be skipped for each sub-block. The BDOF process is not applied to the sub-block when the SAD between the initial L0 predicted samples and the L1 predicted samples is less than a threshold. The threshold is set to equal to (8... W (H>>1), where W indicates the sub-block width and H indicates the sub-block height. To avoid the additional complexity of SAD calculation, the SAD between the initial L0 prediction samples and L1 prediction samples calculated during the DVMR process is reused here.

[0082] Bidirectional optical flow (BDOF) is disabled if BCW is enabled for the current block, meaning the BCW weight index indicates unequal weights. Similarly, BDOF is disabled if WP is enabled for the current block, meaning luma_weight_lx_flag is 1 for either of the two reference images. BDOF is also disabled when the CU is encoded / decoded using symmetric MVD mode or CIIP mode.

[0083] 2.1.1 BDOF in ECM: Sample-based BDOF In sample-based BDOF, motion refinement is not performed on a block-by-block basis (Vx, Vy), but rather on a per-sample basis.

[0084] The encoding / decoding block is divided into 8×8 sub-blocks. For each sub-block, whether to apply BDOF is determined by checking the SAD (Solution-Adjustment Aspect) between two reference sub-blocks against a threshold. If it is decided to apply BDOF to the sub-block, a sliding 5×5 window is used for each sample in the sub-block, and the existing BDOF procedure is applied to derive Vx and Vy for each sliding window. The derived motion refinement (Vx, Vy) is applied to adjust the bidirectional predicted sample values ​​for the center sample of the window.

[0085] 2.2 Decoder-Side Motion Vector Refinement (DMVR) in VVC To improve the accuracy of the motion vector refinement (MV) in the Merge mode, a decoder-side motion vector refinement based on bilateral matching (BM) is applied in the VVC. In the bidirectional prediction operation, a refined MV is searched around the initial MV in reference image lists L0 and L1. The BM method computes the distortion between two candidate blocks in reference image lists L0 and L1. Figure 5 As shown, the SAD between the red blocks of each MV candidate around the initial MV is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bidirectional prediction signal.

[0086] In VVC, the application of DMVR is restricted and can only be used with CUs that are encoded and decoded using the following modes and features: - CU-level Merge pattern with bidirectional prediction of MV.

[0087] - Relative to the current image, one reference image is from the past and another reference image is from the future.

[0088] - The distances from the two reference images to the current image (i.e., the difference in POC) are the same.

[0089] - Both reference images are short-term reference images.

[0090] - The CU has more than 64 luminance samples.

[0091] - Both the CU height and CU width are greater than or equal to 8 luminance samples.

[0092] - The BCW weight index indicates equal weights.

[0093] - WP is not enabled for the current block.

[0094] - CIIP mode is not used in the current block.

[0095] The refined motion vector (MV) derived through the DMVR process is used to generate inter-frame prediction samples and is also used for temporal motion vector prediction in future image encoding and decoding. The original MV is used in the deblocking process and is also used for spatial motion vector prediction in future CU encoding and decoding.

[0096] Additional features of DMVR are mentioned in the following sub-entries.

[0097] In DVMR, the search point revolves around the initial MV, and the MV offset follows the MV difference mirror rule. In other words, any point examined by DMVR (represented by the candidate MV pair (MV0, MV1)) obeys the following two equations.

[0098]

[0099] in This represents the refinement offset between the initial MV and the refined MV in one of the reference images. The refinement search range is two integer luminance samples from the initial MV. The search includes an integer sample offset search phase and a fractional sample refinement phase.

[0100] A 25-point full search is applied to the integer sample offset search. The SAD of the initial MV pair is calculated first. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. To reduce the impact of the uncertainty in DMVR refinement, a bias towards the original MV is proposed during the DMVR process. The SAD between reference blocks referenced by the initial MV candidate references is reduced by 1 / 4 of the SAD value.

[0101] The integer sample search is followed by fractional sample refinement. To save computational complexity, fractional sample refinement is derived using the parametric error surface equation, rather than through an additional search utilizing SAD comparisons. Fractional sample refinement is conditionally invoked based on the output of the integer sample search phase. Fractional sample refinement is further applied when the integer sample search phase terminates in the first or second iteration with the minimum SAD at the center.

[0102] In subpixel offset estimation based on parametric error surfaces, the cost at the center location and the costs at the four nearest neighbor locations are used to fit a two-dimensional parabolic error surface equation of the following form.

[0103] in This corresponds to the score position with the minimum cost, and C corresponds to the minimum cost value. The above equation is solved by using the cost values ​​of the five search points. Calculated as:

[0104] Since all values ​​are positive and the minimum value is ,therefore and The value is automatically constrained to between -8 and 8. This corresponds to a half-pixel offset with 1 / 16 pixel MV precision in VVC. The calculated score It is added to the integer distance thinning MV to obtain the subpixel accurate thinning increment MV.

[0105] In VVC, the resolution of the MV is 1 / 16 of a lumen sample. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search point surrounds the initial fractional pixel MV with an integer sample offset; therefore, for the DMVR search process, samples at those fractional positions need to be interpolated. To reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect of using a bilinear filter is that, utilizing a 2-sample search range, DVMR does not access more reference samples compared to the normal motion compensation process. After obtaining the refined MV through the DMVR search process, a normal 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples than the normal MC process, samples that are not needed by the interpolation process based on the original MV but are needed by the interpolation process based on the refined MV are filled from those available samples.

[0106] When the width and / or height of a CU is greater than 16 luminance samples, it will be further divided into sub-blocks with a width and / or height equal to 16 luminance samples. The maximum cell size for the DMVR search process is limited to 16x16.

[0107] 2.3. Multi-pass decoder-side motion vector refinement (ECM) Multi-pass decoder-side motion vector refinement is applied. In the first pass, bilateral matching (BM) is applied to the codec block. In the second pass, BM is applied to each 16x16 sub-block within the codec block. In the third pass, the motion vectors (MVs) in each 8x8 sub-block are refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for use in spatial and temporal motion vector prediction.

[0108] 2.3.1 First pass – Block-based bilateral matching MV refinement In the first pass, the refined MV is derived by applying the BM to the codec block. Similar to decoder-side motion vector refinement (DMVR), in the bidirectional prediction operation, the refined MV is searched around the two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. The refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MVs based on the minimum bilateral matching cost between the two reference blocks in L0 and L1.

[0109] BM performs a local search to derive the integer sample precision intDeltaMV. The local search applies a 3×3 square search pattern to iterate through the search range [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8.

[0110] The cost of bilateral matching is calculated as: bilCost = mvDistanceCost + sadCost. When the block size is cbW... When cbH is greater than 64, the mean-removed SAD (MRSAD) cost function is applied to remove the DC effect of distortion between reference blocks. The local search of intDeltaMV terminates when bilCost at the center point of the 3×3 search pattern has the minimum cost. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the search for the minimum cost continues until the end of the search range is reached.

[0111] The existing fractional sample refinement is further applied to derive the final deltaMV. The refined MV after the first pass is then derived as: MV0_pass1 = MV0 + deltaMV, MV1_pass1 = MV1 – deltaMV.

[0112] 2.3.2 Second pass – Sub-block based bilateral matching MV refinement In the second pass, the refined MV is derived by applying the BM to 16×16 grid sub-blocks. For each sub-block, the refined MV is searched around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass in the reference image lists L0 and L1. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1.

[0113] For each sub-block, BM performs a full search to derive the integer sample precision intDeltaMV. The full search has a search range [-sHor, sHor] in the horizontal direction and a search range [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8.

[0114] The bilateral matching cost is calculated by applying a cost factor to the SATD cost between the two reference subblocks, as follows: bilCost = satdCost costFactor. Search area (2) sHor + 1) (2) sVer + 1) is divided into Figure 6 A maximum of 5 diamond-shaped search regions are shown. Each search region is assigned a costFactor, which is determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond region is processed sequentially starting from the center of the search region. Within each region, search points are processed in raster scan order from the top left corner to the bottom right corner of the region. When the minimum bilCost within the current search region is less than a threshold (which is equal to sbW), a loss occurs. When sbH), the integer pixel complete search is terminated; otherwise, the integer pixel complete search continues to the next search region until all search points have been checked. Additionally, the search process terminates if the difference between the previous minimum cost and the current minimum cost in an iteration is less than a threshold (which is equal to the area of ​​the block).

[0115] The existing VVC DMVR fractional sample refinement is further applied to derive the final deltaMV(sbIdx2). The MV of the second refinement is then derived as: MV0_pass2(sbIdx2) = MV0_pass1 + deltaMV(sbIdx2) MV1_pass2(sbIdx2) = MV1_pass1 – deltaMV(sbIdx2).

[0116] 2.3.3 Third pass – Sub-block based bidirectional optical flow MV refinement In the third pass, the refined MV is derived by applying BDOF to the 8×8 grid sub-blocks. For each 8×8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy, without clipping from the refined MV of the parent block in the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32.

[0117] The third refined MV (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) is derived as follows: MV0_pass3(sbIdx3) = MV0_pass2(sbIdx2) + bioMv, MV1_pass3(sbIdx3) = MV0_pass2(sbIdx2) – bioMv.

[0118] In all of the aforementioned sub-entries, when surround motion compensation is enabled, the motion vector must be limited to account for surround offset.

[0119] 2.3.4 Adaptive Decoder-Side Motion Vector Refinement The adaptive decoder-side motion vector refinement method is an extension of multi-pass DMVR, consisting of two new Merge modes for refining the motion vector only in one direction (L0 or L1) of the bidirectional predictions of Merge candidates that satisfy the DMVR conditions. The multi-pass DMVR process is applied to the selected Merge candidates to refine the motion vector; however, in the first pass (i.e., PU level) of DMVR, either MVD0 or MVD1 is set to zero.

[0120] The new Merge mode derives its Merge candidates from spatially adjacent encoded / decoded blocks, TMVP, non-adjacent blocks, HMVP, and paired candidates, similar to the regular Merge mode. The difference is that only those satisfying the DMVR conditions are added to the candidate list. Both new Merge modes use the same Merge candidate list. If the BM candidate list contains inherited BCW weights, the DMVR process remains unchanged, except for the use of MRSAD or MRSATD for distortion calculation when the weights are unequal and bidirectional prediction is weighted using BCW weights. The encoding and decoding of the Merge index is the same as in the regular Merge mode.

[0121] 3. Problem Several aspects of BDOF MV refinement / sample adjustment can be improved.

[0122] - The current formula used to drive the BDOF parameters is not an accurate formula.

[0123] - There are no weights to indicate the importance of each sample point in the final formula.

[0124] - There is no filtering process to smooth the final derived MV refinement / sample adjustment.

[0125] - There is no clear distinction regarding the conditions for applying BDOF to refine MV / adjust sample points. Similarly, there is no distinction regarding their formulas.

[0126] 4. Detailed Solution The detailed solutions below should be considered as examples for explaining general concepts. These solutions should not be interpreted in a narrow sense. Furthermore, these solutions can be combined in any way.

[0127] The methods disclosed below can be applied to bidirectional optical flow, decoder-side motion vector refinement, and any of their extensions.

[0128] Derivation of BDOF MV refinement parameters In the following sections, the general equations used to derive the BDOF parameters (vx and vy) are defined as follows:

[0129] Where Gx and Gy represent the sum of the horizontal and vertical gradients of the two reference images, respectively. dI represents the difference between the two reference images. Summation ( Within a predefined region, the predefined region can be an NxM block around the current sample point (used for sample point adjustment BDOF) or an NxM block around the current prediction sub-block (used for MV refinement BDOF).

[0130] 1. A method for deriving gradients, different from BDOF in VVC, is proposed, which can be used to calculate horizontal and / or vertical gradients.

[0131] a. In one example, the gradient is calculated by directly computing the difference between two neighboring samples, i.e.,

[0132] b. In another example, the gradient is calculated by computing the difference between two shifted neighboring samples, i.e.,

[0133] i. shift1 and shift2 can be any integer, such as 0, 1, 2, 6, ..., or even negative integers.

[0134] c. In another example, the gradient can be calculated as a weighted sum of Nb samples before the current sample and Na samples after the current sample:

[0135] i. The weight (i.e., wp) can be any integer, such as -6, 0, 2, 7… or any real number, such as -6.3, -0.77, 0.1, 3.0,… ii. The weights used to calculate the horizontal and vertical gradients can be different from each other.

[0136] (i) Alternatively, the weights used to calculate the horizontal and vertical gradients can be the same.

[0137] iii. Weights can be transmitted from the encoder to the decoder via signals.

[0138] iv. Weights can be derived using the decoded information.

[0139] v. Nb and Na can be any integer, such as 0, 3, 10, ... vi. Nb and Na can be different for calculating the gradient in the horizontal and vertical directions.

[0140] (i) Alternatively, they can be the same for calculating the gradients in both the horizontal and vertical directions.

[0141] vii. In one example,

[0142] (i) Alternatively, additionally, the variable offset can be set to 0 or (1<<(shift-1)).

[0143] 2. A complete linear equation formula is proposed that can be used to derive the final MV refinement.

[0144] a. In one example, after calculating all gradients, s1, s2, s3, s5, and s6 are calculated as explained above:

[0145] i. In one example, in order to derive M The final MV of block N, and (M+K1) around the original block. Samples can be included in the (N+K2) region. For example, K1 and K2 can be any integers, such as 0, 2, 4, 7, 10, ... b. In one example, after calculating all s1, s2, s3, s5, and s6, the determinant values ​​D, Dx, and Dy are calculated as follows: D = (s1 >> shTem) (s5>>shTem) - (s2>>shTem) (s2>>shTem), Dx = (s3>>shTem) (s5>>shTem) - (s6>>shTem) (s2>>shTem), Dy = (s1>>shTem) (s6>>shTem) - (s3>>shTem) (s2>>shTem).

[0146] i. In one example, shTem can be any integer, such as 0, 1, 3, ... c. In one example, after calculating D, Dx, and Dy, vx and vy can be derived as: vx = Dx / D and vy = Dy / D.

[0147] i. In another example, if abs(D) is less than a predefined threshold C, then vx and vy are set to zero. C can be any non-negative number, such as 0, 10, 17, ... d. In one example, shifting and limiting of any amount can be involved to derive the final vx and vy.

[0148] i. In one example, the numerator and / or denominator may have an additional shift, such that they are shifted left by K overall, resulting in higher precision for the final derived vx and vy. K can be any integer, such as 0, 1, 3, 4, 6, ... ii. In one example, these shifts can occur in any order, such as having a shift at the beginning, and / or having a shift for intermediate variables and / or having a shift on the final MV.

[0149] iii. In one example, the final vx and vy can be bounded between -B and B, where B can be any integer, such as 2, 10, 17, 32, 100, 156, 725, ... e. In one example, the final vx and vy can be multiplied (or similarly divided) by a number before being used in the motion compensation process.

[0150] i. In one example, vx and vy can be multiplied by R, where R is any real number, such as 1.25, 2, 3.1, 4, ... ii. In another example, vx and vy can be divided by R, where R is any real number, such as 1.25, 2, 3.1, 4, ... iii. In one example, the values ​​of the numbers to be multiplied (or divided) by the final vx and vy can be different for vx and vy.

[0151] iv. In one example, the value of the number to be multiplied (or divided) by the final vx, vy can depend on the block size, sequence resolution, block characteristics, etc.

[0152] 3. It is proposed that the solutions to some linear equations can be used to derive the final MV refinement.

[0153] a. In one example, after calculating all gradients, s1, s2, s3, s5, and s6 are calculated as explained above:

[0154] b. In one example, after calculating all s1, s2, s3, s5, and s6, approximate versions of vx and vy can be calculated as: vx = s3 / s1, vy = (s6 – s2 vx) / s5.

[0155] c. In another example, after calculating vx similarly to the above, a portion of vx can be put into the second formula to derive vy.

[0156] i. In one example, vy can be derived as vy = (s6 – s2) vx / T) / s5, where T can be any real number, such as 1.1, 2, 4, ... d. In one example, after calculating all s1, s2, s3, s5, and s6, approximate versions of vx and vy can be calculated as: Assuming vx is zero: vy = s6 / s5, Insert vy into the first formula: vx = (s3 – s2) vy) / s1.

[0157] e. In another example, after calculating vy similar to the above, a portion of vy can be inserted into the second formula to derive vx.

[0158] i. In one example, vx can be derived as vx = (s3 – s2) vy / T) / s1, where T can be any real number, such as 1.1, 2, 4, ... f. In one example, after calculating all s1, s2, s3, s5, and s6, approximate versions of vx and vy can be calculated as: Assume vy is zero: vx = s3 / s1.

[0159] Assuming vx is zero: vy = s6 / s5.

[0160] 4. A simplified solution is proposed that can be used to derive the final MV refinement.

[0161] a. In one example, the methods for VVC BDOF explained in the background section can be used to derive approximate versions of s1, s2, s3, s5, and s6.

[0162] b. In one example, after calculating approximate versions of s1, s2, s3, s5, and s6, the determinant values ​​D, Dx, and Dy are calculated as follows: D = (s1 >> shTem) (s5>>shTem) – (s2>>shTem) (s2>>shTem), Dx = (s3>>shTem) (s5>>shTem) – (s6>>shTem) (s2>>shTem), Dy = (s1>>shTem) (s6>>shTem) – (s3>>shTem) (s2>>shTem).

[0163] i. In one example, after calculating D, Dx, and Dy; vx and vy can be derived as: vx = Dx / D and vy = Dy / D.

[0164] ii. In one example, shTem can be any integer, such as 0, 1, 3, ... c. In one example, after calculating approximate versions of s1, s2, s3, s5, and s6, approximate versions of vx and vy can be calculated as follows: Assuming vy is zero: vx = s3 / s1, Substituting vx into the second formula: vy = (s6 – s2) vx) / s5.

[0165] i. Alternatively, the modified vx can be inserted into the second formula: vy = (s6 – s2 vx / T) / s5, where T can be any real number, such as 1.1, 2, 4, ... ii. Alternatively, firstly, vx can be assumed to be zero, and vy can be derived, then vy or its scaled version can be inserted into the first equation, and vx can be derived.

[0166] 5. Any combination of the methods described above can be used to derive the final MV refinement.

[0167] a. In one example, any combination of the methods (2, 3, and 4) explained above can be combined and used together.

[0168] Derivation of BDOF sample point adjustment parameters 6. Any of the methods for BDOF MV refinement explained above can also be used to derive BDOF sample adjustment parameters.

[0169] a. In one example, after calculating all gradients, s1, s2, s3, s5, and s6 are calculated as explained above:

[0170] i. In one example, samples within a KxK region surrounding a sample point can be included in the derivation. K can be any integer, such as 1, 3, 4, 5, 7, 10, ... b. In one example, after calculating s1, s2, s3, s5, and s6, the determinant values ​​D, Dx, and Dy are calculated as follows: D = (s1 >> shTem) (s5>>shTem) - (s2>>shTem) (s2>>shTem), Dx = (s3>>shTem) (s5>>shTem) - (s6>>shTem) (s2>>shTem), Dy = (s1>>shTem) (s6>>shTem) - (s3>>shTem) (s2>>shTem).

[0171] i. shTem can be any integer, such as 0, 1, 3, ... ii. In one example, after calculating D, Dx, and Dy; vx and vy can be derived as: vx = Dx / D and vy = Dy / D.

[0172] iii. In another example, if abs(D) is less than a predefined threshold C, then vx and vy are set to zero. C can be any non-negative number, such as 0, 10, 17, ... c. In one example, after calculating s1, s2, s3, s5, and s6, approximate versions of vx and vy can be calculated as: vx = s3 / s1, vy = (s6 – s2 vx) / s5.

[0173] i. Alternatively, the modified vx can be included in the second formula: vy = (s6 – s2 vx / T) / s5, where T can be any real number, such as 1.1, 2, 4, ... ii. Alternatively, firstly, vx can be assumed to be zero, and vy can be derived, then vy or its scaled version can be substituted into the first equation, and vx can be derived.

[0174] d. In one example, the methods for VVC BDOF explained in the background section can be used to derive approximate versions of s1, s2, s3, s5, and s6.

[0175] e. In one example, the final vx and vy can be multiplied (or divided or shifted) by a number before being used in the sample adjustment process.

[0176] i. In one example, vx and vy can be multiplied by R, where R is any real number, such as 1.25, 2, 3.1, 4, ... ii. In another example, vx and vy can be divided by R, where R is any real number, such as 1.25, 2, 3.1, 4, ... iii. In one example, the values ​​of the numbers to be multiplied (or divided) by the final vx and vy can be different for vx and vy.

[0177] iv. In one example, the value of the number to be multiplied (or divided) by the final vx, vy can depend on the block size, sequence resolution, block characteristics, position in the block, etc.

[0178] Regarding the application of weights in parameter derivation 7. It was proposed that any weights can be applied before adding BDOF intermediate parameters for MV refinement.

[0179] a. In one example, during the addition of parameters to obtain s1, s2, s3, s5, and s6, in the target region (M_ext around the current block) Within the N_ext region, all values ​​are added with similar weights (1).

[0180] b. In another example, during the addition of parameters to obtain s1, s2, s3, s5, and s6, in the target region (M_ext around the current block) These values, when multiplied within the N_ext region, depend on their position within the extended block (target region). The predefined weights of the positions in the () are added afterward.

[0181] c. In one example, these predefined weights are defined as follows: w = (x>= (width / 2)?width - x : x + 1) (y>= (height / 2)?height-y:y+1) For x from 0 to width - 1 and y from 0 to height - 1.

[0182] Width and height represent the width and height of the target area.

[0183] d. In another example, these predefined weights can be generated using some known probability distribution, such as the standard deviation with any value ( = 1, 1.5, 4 or any other real number) and a Gaussian distribution at the center location.

[0184] i. In one example, such as Figure 7 As shown, these weights utilize the 12x12 region. A Gaussian distribution with a value of 2.5 was generated.

[0185] ii. In one example, such as Figure 8 As shown, these weights utilize the 12x12 region. A Gaussian distribution with a value of 4 was generated.

[0186] e. In another example, during the addition of parameters to obtain s1, s2, s3, s5, and s6, in the target region (M_ext around the current block) Within the N_ext region, these values ​​are utilized depending on their location within the extended block (target region). The predefined value of the position in ) is shifted and then added.

[0187] f. In one example, the weight matrix can be represented as a left (or right) shift matrix, and depending on the matrix entries, the data is shifted (left or right) before summing.

[0188] 8. In one example, different weights can be applied based on block size, block shape, block characteristics, sequence resolution, etc.

[0189] i. Alternatively, no weights can be applied based on block size, block shape, block characteristics, sequence resolution, etc.

[0190] ii. The weight matrix can be explicitly encoded or decoded in the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), or Strip Header (SH).

[0191] 9. It was proposed that any weights can be applied before adding BDOF intermediate parameters for sample adjustment.

[0192] a. In one example, during the addition of parameters to obtain s1, s2, s3, s5, and s6, in the target region (K1 around the current sample point) Within region K2, all values ​​are added with similar weights (1). K1 and K2 can be any integers, such as 1, 2, 3, 5, 8, ... b. In another example, during the addition of parameters to obtain s1, s2, s3, s5, and s6, in the target region (K1 around the current sample point) Within the K2 region, these values ​​are multiplied depending on their position within the extended block (target region). The predefined weights of the positions in the () are added afterward.

[0193] c. In one example, these predefined weights are defined as follows: w = (x>= (K1 / 2) ? K1 - x : x + 1) (y>= (K2 / 2) ? K2 - y : y + 1), For x from 0 to K1 - 1 and y from 0 to K2 – 1.

[0194] K1 and K2 represent the width and height of the target area.

[0195] d. In another example, these predefined weights can be generated using some known probability distribution, such as the standard deviation with any value ( = 1, 1.5, 2, 4 or any other real number) and a Gaussian distribution with any central location.

[0196] i. In one example, such as Figure 9 As shown, these weights utilize the 5x5 area A Gaussian distribution with a value of 1 is generated.

[0197] ii. In one example, such as Figure 10 As shown, these weights utilize the 5x5 area A Gaussian distribution with a value of 2 is generated.

[0198] e. In one example, the weight matrix can be represented as a left (or right) shift matrix, and depending on the matrix entries, the data is shifted (left or right) before summing.

[0199] f. In one example, different weights can be applied based on block size, block shape, block characteristics, sequence resolution, etc.

[0200] i. Alternatively, no weights can be applied based on block size, block shape, block characteristics, sequence resolution, etc.

[0201] Regarding the application of filters for final MV refinement or sample adjustment 10. It is proposed that any type of filter can be applied to the final derived MV refinement (vx and vy). Some examples are provided in... Figure 11 It is depicted in the middle.

[0202] a. In one example, any smoothing filter of any shape can be applied to all MVs derived by BDOF for each sub-block.

[0203] b. In one example, all MVs inside the PU can be used during filter application.

[0204] c. In another example, during the filter application, only the second round MV with similar DMVR can be used for those MVs.

[0205] d. In one example, a shape filter with any weights can be applied to the MV.

[0206] i. In one example, the weight of the center can be 8, and the weights of the four edges can be 1.

[0207] ii. In one example, the center weight can be 4, and the weights of the four edges can be 1.

[0208] iii. In one example, the center weight can be 4, and the weights of the four edges can be 2.

[0209] iv. In one example, the center weight can be 4, and the weights of the four edges can be 3.

[0210] v. In one example, the weight of the center can be 1, and the weights of the four edges can be 1.

[0211] 11. It is proposed that any type of filter can be applied to the final derived BDOF sample MV adjustment or final sample adjustment. Some examples are provided in... Figure 11 It is depicted in the middle.

[0212] a. In one example, the filter is applied to all (vx, vy) or finally adjusted within the sub-block.

[0213] b. In one example, a shape filter with any weights can be applied to (vx, vy) or ultimately adjusted.

[0214] i. In one example, the weight of the center can be 8, and the weights of the four edges can be 1.

[0215] ii. In one example, the center weight can be 4, and the weights of the four edges can be 1.

[0216] iii. In one example, the center weight can be 4, and the weights of the four edges can be 2.

[0217] iv. In one example, the center weight can be 4, and the weights of the four edges can be 3.

[0218] v. In one example, the weight of the center can be 1, and the weights of the four edges can be 1.

[0219] Conditions for applying BDOF 12. It proposes conditions that can exist when applying BDOF MV refinement or BDOF sample adjustment.

[0220] a. In one example, the conditions for applying BDOF MV refinement can be similar to those for applying BDOF sample adjustment.

[0221] b. In another example, the conditions for applying BDOF MV refinement may differ from the conditions for applying BDOF sample adjustment. For instance, BDOF MV refinement may be applied to bidirectional predictive codecs with unequal weights, while BDOF sample adjustment may be applied only to bidirectional predictive codecs with equal weights.

[0222] 13. It is proposed that the cost for evaluating BDOF conditions can depend on the cost between two reference image blocks.

[0223] a. In one example, different cost functions can be used to derive the cost.

[0224] i. In one example, the cost could be the sum of absolute differences (SAD) between two reference image patches.

[0225] ii. In one example, the cost could be the sum of absolute transformation differences (SATD) between two reference image patches or any other cost measure.

[0226] iii. In one example, the cost could be the sum of absolute differences between two reference image patches based on mean removal (MR-SAD).

[0227] iv. In one example, the cost could be a weighted average of SAD / MR-SAD and SATD between two reference image patches.

[0228] v. In one example, the cost function between two reference image patches could be: (i) Sum of absolute differences (SAD) / SAD after removing the mean (MR-SAD); (ii) Sum of absolute transformation differences (SATD) / SATD with mean removed (MR-SATD); (iii) Sum of squared differences (SSD) / SSD after mean removal (MR-SSD); (iv) SSE / MR-SSE; (v) Weighted SAD / Weighted MR-SAD; (vi) Weighted SATD / Weighted MR-SATD; (vii) Weighted SSD / Weighted MR-SSD; (viii) Weighted SSE / Weighted MR-SSE; (ix) Gradient information.

[0229] Regarding BDOF MV refined sub-block size 14. It is proposed that any sub-block size, depending on the conditions, can be used as the BDOF MV refined sub-block size.

[0230] a. In one example, the sub-block size can be a fixed size, such as N x M, where N and M can be any positive integer, such as 1, 2, 3, 4, 5, 8, 12, 32, ... b. In another example, the sub-block size can depend on the current PU or CU size. For example, for a block size WxH, a sub-block size W1xH1 can be used, where W1 and H1 depend on W and H, and can be any positive integer.

[0231] i. In one example, for the number of samples between C_i and C_(i+1) (i.e., width multiplied by height (W) For a block H), the sub-block size W_i x H_i can be used. C_i can be any non-negative number, such as 0, 4, 20, 128, 256, 951, 2048, 4100, ..., and W_i x H_i can be any positive integer pair, such as 2x2, 4x4, 8x4, 4x8, 8x8, 16x16, 19x15, ... ii. In one example, for a block having a width W between Cw_i and Cw_(i+1) and a height H between Ch_j and Ch_(j+1), the sub-block size W_i x H_j can be used. Cw_i and Ch_j can be any non-negative numbers, such as 0, 4, 20, 128, 256, 951, 2048, 4100, ..., and W_i x H_j can be any positive integer pair, such as 2x2, 4x4, 8x4, 4x8, 8x8, 16x16, 19x15, ... c. In one example, the sub-block size may depend on the color components and / or color format.

[0232] d. In one example, the size of a sub-block can depend on the encoded / decoded information of the current block.

[0233] i. In one example, the encoded and decoded information is residual information.

[0234] ii. In one example, the encoded / decoded information is the encoding / decoding tool applied to the current block.

[0235] e. In one example, the sub-block size can depend on the information of the predicted block.

[0236] f. In one example, the size of the sub-block may depend on the characteristics of the reference image.

[0237] i. In one example, the sub-block size can be determined by the similarity of two predicted values ​​from two reference images. If the two predicted values ​​are similar, such as having a smaller SAD between them, a larger sub-block size can be applied; otherwise, a smaller sub-block size can be applied.

[0238] ii. In one example, the sub-block size can be determined by the distribution of the difference between two predictions. Those sub-blocks with difference energy (such as SAD or SSE) can be merged into larger cells for MV refinement, thereby reducing computational complexity.

[0239] g. In one example, the sub-block size can depend on the temporal gradients of two reference blocks.

[0240] i. In one example, any cost function (such as SAD) can be used to compute the gradient (or difference) of two reference blocks.

[0241] h. In one example, the spatial gradient of a reference block can be used to determine the size of a sub-block.

[0242] i. In one example, the sub-block size can depend on the quantization parameter (qp) value.

[0243] i. In one example, for qp smaller than X, the sub-block size W_X x H_X can be used.

[0244] ii. In one example, for qp greater than X, the sub-block size W_X x H_X can be used.

[0245] iii. In one example, for qpX, the sub-block size W_X x H_X can be used.

[0246] iv. In one example, X can be any non-negative integer, such as 10, 22, 27, 32, 37, 42, ..., and W_X and H_X can be any positive integer, such as 1, 2, 3, 4, 8, 10, ... v. In one example, qp can be the qp of the current CU, the qp of the current stripe, or the qp of the entire sequence.

[0247] vi. In one example, the decision regarding the sub-block size can be made by the encoder, and it may or may not be signaled to the decoder. Similarly, it can be made by the decoder.

[0248] vii. In one example, increasing or decreasing the sub-block size based on qp can be determined by either the encoder or the decoder.

[0249] j. In one example, the sub-block size can depend on the prediction type.

[0250] k. In one example, the sub-block size may depend on the DMVR first and / or second stage adjustment values.

[0251] l. In one example, the sub-block size can depend on the sequence resolution.

[0252] m. In one example, the sub-block size can depend on the encoding / decoding tool applied to the current block.

[0253] n. In one example, the sub-block size can depend on the temporal layer.

[0254] i. In one example, for a time-domain layer between Ti and Tj, the sub-block size W_ij x H_ij can be used. Ti and Tj can be any non-negative integers, such as 0, 1, 3, 4, ..., and W_ij and H_ij can be any positive integers, such as 2, 4, 6, 16, ... o. In one example, the sub-block size can be a function of all or some of the parameters mentioned above.

[0255] p. In the above examples, the sub-block size of the luminance and / or chroma blocks can be determined based on the examples above.

[0256] i. Alternatively, the sub-block size of the chroma block can be derived based on the sub-block size of the luma block and the color format and / or whether individual planar codecs are enabled.

[0257] Regarding asymmetric BDOF 15. It was proposed that the MV adjustment for the first and second lists does not have to be symmetrical.

[0258] a. In one example, the MV refinement for reference image 0 could be (vx0, vy0), and the MV refinement for reference image 1 could be (-vx1, -vy1), where vx0, vy0, vx1, and vy1 can be any real numbers or integers. They may or may not have a relationship together.

[0259] b. In one example, the general equations used to derive vx0, vy0, vx1, and vy1 can be written as the following four equations:

[0260] Where Gx0, Gx1, Gy0, and Gy1 represent the horizontal gradient of reference image 0, the horizontal gradient of reference image 1, the vertical gradient of reference image 1, and the vertical gradient of reference image 2, respectively. dI represents the difference between the two reference images. Summation ( Within a predefined region, the predefined region can be an NxM block around the current sample point (used for sample point adjustment BDOF) or an NxM block around the current prediction sub-block (used for MV refinement BDOF).

[0261] c. Alternatively, in matrix format they can be written as:

[0262] The parameters in the matrix format match the parameters in the equation.

[0263] d. In one example, the general formula for determinants can be used to solve the above linear equations.

[0264] e. In one example, Gaussian elimination can be used to solve the linear equation above.

[0265] f. In one example, any other method (including matrix factorization) can be used to solve the above linear equation.

[0266] g. In one example, vx1 can be equal to k. vx0, and vy1 can be equal to k vy0, and k can be any real number or integer, such as -0.3, 0, 0.1, 2, 3, ... i. In one example, any nonlinear method can be used to derive and solve nonlinear equations.

[0267] h. In one example, any weighted sums described in the previous section can be used for summation.

[0268] i. In one example, asymmetric BDOF can be applied to BDOF MV refinement and BDOF sample adjustment.

[0269] j. In one example, asymmetric BDOF can be applied only to BDOF MV refinement.

[0270] k. In one example, asymmetric BDOF can be applied only to BDOF sample adjustment.

[0271] l. In one example, whether and / or how to apply asymmetric BDOF may depend on the POC or at least one POC distance.

[0272] i. In one example, whether and / or how to apply asymmetric BDOF may depend on |POC_ref0-POC_cur| and / or |POC_ref1-POC_cur|, where POC_ref0 and POC_ref1 represent the POCs of two reference images, and POC_cur is the POC of the current image.

[0273] m. In one example, whether and / or how to apply asymmetric BDOF can depend on the BCW weights.

[0274] n. In one example, whether and / or how asymmetric BDOF is applied may depend on at least one template of the current block.

[0275] i. Furthermore, whether and / or how asymmetric BDOF is applied may depend on at least one reference template of the template of the current block.

[0276] Conditions for applying BDOF and its combination with other tools 16. It was proposed that BDOF and / or asymmetric BDOF (MV refinement or sample adjustment or both) can be used in combination with other tools or excluded from the use of other tools.

[0277] a. In one example, BDOF can be applied to blocks encoded and decoded with unequal BCW weights.

[0278] i. In one example, BDOF can be applied using BCW weights from a predefined set (such as {3}, or {3, 5} or {-1, 3}).

[0279] b. In one example, BDOF can be applied to blocks of two reference images on the same side of the current frame.

[0280] c. In one example, BDOF can be applied to a block of a reference image on the opposite side of the current frame.

[0281] i. In one example, they can be at the same distance from the current frame.

[0282] ii. In another example, they may have a different distance from the current frame.

[0283] d. In one example, BDOF can be applied in combination with LIC.

[0284] i. Alternatively, if the block uses a LIC, it can be turned off.

[0285] e. In one example, BDOF can be applied in combination with OBMC.

[0286] i. Alternatively, if the block uses OBMC, it can be turned off.

[0287] f. In one example, BDOF can be applied in combination with CCIP.

[0288] i. Alternatively, it can be turned off if the block uses CIIP.

[0289] g. In one example, BDOF can be applied in combination with SMVD.

[0290] i. Alternatively, it can be turned off if the block uses SMVD.

[0291] h. In one example, the BDOF DMVR or BDOF - like taps can be controlled individually. For example, the BDOF DMVR is applied, but the BDOF - like taps are not applied.

[0292] i. The control can be at the sub - PU level, or CU level, or CTU level.

[0293] Regarding merging sub-blocks with similar MVs before applying BDOF DMVR or BDOF samples. 17. It is proposed that multiple sub - blocks can share the same MV after / before the MV refinement process such as BDOF or DMVR.

[0294] a. In one example, it is proposed to check for similar MVs of neighboring sub - blocks and combine them before applying the BDOF DMVR or BDOF - like taps process.

[0295] b. In one example, multiple sub - blocks sharing the same MV can perform motion compensation as a whole.

[0296] c. In one example, N1 neighboring sub - blocks in a row with similar MVs can be merged.

[0297] i. N1 can be any integer, such as 2, 3, 4, 10, … d. In one example, N2 neighboring sub - blocks in a column with similar MVs can be merged.

[0298] i. N2 can be any integer, such as 2, 3, 4, 10, … e. In one example, all neighboring sub - blocks in a row with similar MVs up to the r - th round (first, second, …) of the DMVR sub - PU boundary can be merged. These sub - PU boundaries can occur every K pixels, where K can be any integer, such as 8, 16, 19, 32, … f. In one example, all neighboring sub - blocks in a column with similar MVs up to the r - th round (first, second, …) of the DMVR sub - PU boundary can be merged. These sub - PU boundaries can occur every K pixels, where K can be any integer, such as 8, 16, 19, 32, … g. In one example, the decision for row - based or column - based merging can depend on the width (W) and / or height (H) of the block size.

[0299] i. In one example, if W >= H, a row - based merging method can be used. Otherwise, a column - based merging method can be used.

[0300] ii. Alternatively, if W < H, a row - based merging method can be used. Otherwise, a column - based merging method can be used.

[0301] h. In one example, the MV checking and merging process can be applied to each sub-block within a larger M×N sub-block. M and N can be any integers, such as 4, 5, 10, 16, 32, ... i. In one example, all 4x4 (or 8x8) sub-blocks within 16x16, 8x8, 16x8, or 32x32 can be merged.

[0302] ii. In one example, these M and N can be variable or fixed.

[0303] i. In one example, all sub-blocks within a PU or CU can be merged.

[0304] j. In one example, sub-blocks with nearly similar (but not necessarily identical) MVs can also be merged in all the above scenarios. The criterion for nearly similarity can be defined as whether the first or second Euclidean distance of the MVs is less than a threshold. The merged MV can be the average, mode, etc. of all MVs. Or it can be the MV in the center, top left, bottom right, or other positions.

[0305] k. Alternatively, additionally, when two sub-blocks have similar motions, the motion information of one or both sub-blocks can be modified before use so that the two sub-blocks will use the same motion to perform the operation performed earlier.

[0306] l. In one example, two motions are considered similar motions when they use the same reference image and their video images (MVs) are similar (e.g., the MV difference is less than a threshold).

[0307] m. The above examples can be applied to each prediction direction.

[0308] i. Alternative locations, which can be applied together for all predicted directions.

[0309] Regarding parallelization, code optimization, and applying shifts to BDOF. 18. Parallelization was proposed to compute BDOF parameters.

[0310] a. In one example, the SIMD implementations of all relevant functions can be used.

[0311] b. In one example, during the calculation of the sum of parameters for BDOF samples (or DMVR), the K sums of the sample sums at one iteration can be derived. K can be any integer, such as 2, 3, 4, 5, 8, ... c. In one example, the weighted sum can be implemented as a proper left shift.

[0312] i. In one example, the multiplication using weight w_i can be replaced by a left shift of log2(1+w_i).

[0313] 19. Several code optimizations for BDOF are proposed.

[0314] a. In one example, the BDOF DMVR parameter will not always be calculated. Its calculation can be delayed and conditional if actually needed.

[0315] i. In one example, it is only calculated if no BDOF samples are applied.

[0316] b. In one example, the BDOF sample parameters will not be calculated continuously. If necessary, their calculation can be delayed and conditional.

[0317] i. In one example, it is only calculated if the BDOF DMVR phase does not result in an MV update.

[0318] ii. In one example, it is only calculated if BDOF DMVR is not applied.

[0319] c. In one example, for a block where BDOF DMVR is off, different sub-block sizes can be used to check the application conditions of BDOF samples.

[0320] i. In one example, the new sub-block size can be larger or smaller than the BDOF DMVR sub-block size. It can be M×N, where M and N can be any integer. Here are some examples for M×N sizes: 2×2, 4×4, 4×8, 8×4, 8×8, ... 20. It is proposed to add shifts (right shift or left shift) at different stages of BDOF parameter derivation in order to remove noise, avoid overflow, reduce data bandwidth, or improve the accuracy of the derived parameters.

[0321] a. Shift operations can be right shifts or left shifts.

[0322] b. Offsets can be added before and / or after the shift operation.

[0323] c. The result of the shift can be limited to a range.

[0324] d. In one example, the data can be shifted by Shift1 before / after the gradient is calculated. Shift1 can be any right or left shift with any integer value such as 0, 1, 3, 4, 6, ...

[0325] e. In one example, the data can be shifted by Shift2 before / after calculating the difference in brightness. Shift2 can be any right or left shift with any integer value such as 0, 1, 3, 4, 6, ...

[0326] f. In one example, the data can be shifted by Shift3 before / after multiplying by the gradient or brightness difference. Shift3 can be any right or left shift with any integer value such as 0, 1, 3, 4, 6, ...

[0327] g. In one example, data can be shifted by Shift4 before / after the sum of the calculated parameters. Shift4 can be any right or left shift with any integer value such as 0, 1, 3, 4, 6, ...

[0328] h. In one example, data can be shifted by Shift5 before / after calculating the determinant (multiplication of the final parameters). Shift5 can be any right or left shift with any integer value such as 0, 1, 3, 4, 6, ...

[0329] i. In one example, the data can be shifted by Shift6 before / after the division of the determinant to obtain the final scaled MV adjustment. Shift6 can be any right or left shift with any integer value such as 0, 1, 3, 4, 6, ...

[0330] j. In one example, the data can be shifted by Shift7 before / after the calculation of sample point adjustments. Shift7 can be any right or left shift with any integer value such as 0, 1, 3, 4, 6, ...

[0331] k. In one example, the shift parameter can depend on the bit depth.

[0332] Iterations on the application of BDOF DMVR 21. It was proposed that BDOF DMVR or BDOF sample refinement can be applied in an iterative manner.

[0333] a. In one example, there can be N iterations for applying BDOF DMVR or BDOF sample refinement, where N can be any non-negative integer, such as 0, 1, 2, 3, 5, 8, ... b. In one example, the refined samples and / or MVs in one iteration of BDOF DMVR or BDOF sample refinement can be applied and used to derive the refinement in the next iteration of BDOF DMVR or BDOF sample refinement.

[0334] c. In one example, the number of iterations can depend on the PU / CU / subPU block size, qp, neighboring blocks, temporal layers, and / or all combinations thereof. Typically, it can depend on the conditions explained in item 14 above.

[0335] d. In one example, whether to apply another round of iterations may depend on the results of previous iterations. In one embodiment, the iteration stops if the refinement is less than a predefined threshold. In another embodiment, the iteration stops if the ratio between the refinement at the current iteration and the refinement at the previous iteration is less than a predefined threshold.

[0336] e. In one example, each iteration can be independent of each other, and the decision to apply another round of iterations may not depend on the results of previous rounds.

[0337] f. In one example, the number of iterations can depend on the sequence resolution.

[0338] g. In one example, the number of iterations can depend on the block size.

[0339] h. In one example, the number of iterations can depend on qp.

[0340] i. In one example, the number of iterations can depend on any of the items explained in #14 above.

[0341] j. In one example, the iterative BDOF DMVR or BDOF sample refinement process can be terminated after N rounds of BDOF DMVR or BDOF sample refinement have been performed.

[0342] k. In one example, the iterative BDOF DMVR or BDOF sample refinement process can be terminated when certain conditions are met.

[0343] i. In one example, the condition is that the cost is below (or not greater than) a threshold.

[0344] ii. The cost can be defined as the SAD (or SATD or SSD or MR-SAD) between two predictions from two lists.

[0345] l. In one example, for applying BDOF DMVR to each sub-block in each iteration, the threshold can be fixed or can be different (for each iteration).

[0346] i. This threshold can be compared to a cost defined between the two predictions from each list. This cost can be the SAD cost between their pixels.

[0347] m. The number of iterations can be transmitted via signaling at the sequence level (SPS), the image level, or the strip header.

[0348] The number of iterations can depend on the temporal level of the current image.

[0349] 22. It was proposed that the size of the sub-blocks in each iteration can be the same or different.

[0350] a. In one example, all iterations have the same sub-block size.

[0351] i. This same size can be fixed for all conditions.

[0352] ii. As explained in item 14 above, the same size can be an adaptive size.

[0353] b. In one example, iteration j can have a sub-block size W_j. H_j, where j can be any non-negative integer, and W_j and H_j can be any positive integer, such as 2, 4, 8, 11, 16.

[0354] c. In one example, the size of the sub-block in each iteration may depend on the MV derived from the previous iteration.

[0355] d. In one example, the sub-block size may depend on the sequence resolution and / or a combination of any other factors.

[0356] 23. It is proposed that the derived MV can be adjusted with the same or different scaling in each iteration. This scaling is multiplied by the derived MV.

[0357] a. In one example, all iterations can have the same scaling factor s, where s is any real number, such as 0.25, 0.4, 0.5, 1, 1.63, 2, 4, ... This s is multiplied by the derived BDOF MV.

[0358] i. The same scaling factor can be fixed for all conditions.

[0359] ii. The same scaling factor can be adapted based on what is explained in item 14 above.

[0360] b. In one example, each iteration can have its own scaling factor. For example, iteration j will have a scaling factor s_j, where s_j can be any real number, such as 0.25, 0.4, 0.5, 1, 1.63, 2, 4, ...

[0361] c. In one example, the scaling factor for each iteration may depend on the MV derived from previous and / or current iterations.

[0362] i. In one example, it can depend on the angle between the derived MVs.

[0363] ii. In one example, it can depend on the size of the MV.

[0364] iii. In one example, it can depend on the qp value.

[0365] iv. In one example, it can depend on a combination of multiple factors, such as MV size, MV angle, sub-block size, qp, block size, ...

[0366] d. In one example, a scaling factor, ss, may be available for the BDOF samples, depending on the number of iterations and / or the derived MV adjustment and / or other conditions described in item 14. ss can be any real number, such as 0.2, 0.25, 0.5, 1.1, ...

[0367] e. In one example, scaling may depend on the sequence resolution and / or a combination of any other factors.

[0368] DMVR, BDOF sampling points for cases with unequal POC distances 24. It is proposed that unequal POC distances can use DMVR and / or BDOF sampling points.

[0369] a. In one example, the case of unequal POC distances can be handled using only DMVR (without the BDOF portion).

[0370] b. In one example, the case of unequal POC distances can be handled by using only BDOF DMVR (without BDOF samples and other parts of the DMVR).

[0371] c. In one example, the case of unequal POC distances can be solved using only BDOF samples.

[0372] d. In one example, the case of unequal POC distances can be handled using only DMVR (with the BDOF DMVR portion).

[0373] e. In one example, the case of unequal POC distances can be handled by using only BDOF DMVR and BDOF samples (without the other parts of DMVR).

[0374] f. In one example, the case of unequal POC distances can use all DMVRs (including BDOF DMVRs) and BDOF samples.

[0375] g. In one example, BDOF DMVR and / or BDOF samples can use the same formulas described in Sections 1-6 above.

[0376] h. In another example, BDOF DMVR and / or BDOF samples can use the updated formula:

[0377] Where Gx and Gy represent the sum of the horizontal and vertical gradients over the two reference images, respectively, weighted by t0 and t1. t0 and t1 represent the point of convergence (POC) distances from the current frame to reference frames 0 and 1, respectively. The final adjustment for list 0 will be (t0... vx, t0 vy), and the final adjustment for list 1 will be (-t1) vx, -t1 vy).

[0378] 25. It was proposed that whether and / or how to perform MV scaling (also referred to as “scaling”) in DMVR can depend on the POC distance of at least two reference images.

[0379] a. In one example, different scaling can be used for cases with unequal POC distances, where the POC distances of the two reference images are different.

[0380] b. In one example, scaling may not be involved in the first round of DMVR (PU, CU level), and both List 0 and List 1 use the exact same final MV adjustment with mirror properties.

[0381] c. In one example, for the first round of DMVR (PU, CU level), some scaling may be involved, and list 0 and list 1 use different final MV adjustments. However, this scaling may or may not be applied during bilateral matching cost calculation.

[0382] d. In one example, list 0 can be adjusted using the derived MV, and list 1 can be a scaled version of the MV adjustment of list 0. This scaling can be proportional to the POC distance of list 1 to the current frame and the POC distance of list 0 to the current frame.

[0383] e. In one example, list 1 can be adjusted using the derived MV, and list 0 can be a scaled version of the MV adjustment of list 1. This scaling can be proportional to the POC distance of list 0 to the current frame and the POC distance of list 1 to the current frame.

[0384] f. In one example, a list with a shorter POC distance can be adjusted using the derived MV, and another list can be adjusted using a scaled-up version of the first MV. This scaling can be proportional to the difference in POC distance.

[0385] g. In one example, a list with a longer POC distance can be scaled using the derived MV, and another list can be scaled using a smaller version of the first MV. This scaling can be proportional to the difference in POC distance.

[0386] h. In one example, the scaling values ​​in all the above paragraphs can be capped to predefined minimum and maximum values. This scaling can be exactly the POC distance ratio or its capped version.

[0387] i. In one example, in the second round of DMVR (sub-PU level), scaling may not be involved, and both List 0 and List 1 use the exact same final MV adjustment with mirror properties.

[0388] j. In one example, for the second round of DMVR (sub-PU level), some scaling may be involved, and list 0 and list 1 use different final MV adjustments. However, this scaling may or may not be applied during bilateral matching cost calculation.

[0389] k. In one example, for the second round of DMVR, list 0 can use the derived MV adjustment, and list 1 can use a scaled version of the MV adjustment of list 0. This scaling can be proportional to the POC distance of list 1 to the current frame and the POC distance of list 0 to the current frame.

[0390] l. In one example, for the second round of DMVR, list 1 can use the derived MV adjustment, and list 0 can use a scaled version of the MV adjustment of list 1. This scaling can be proportional to the POC distance of list 0 to the current frame and the POC distance of list 1 to the current frame.

[0391] m. In one example, for the second round of DMVR, the list with shorter POC distances can be adjusted using the derived MV, and the other list can be adjusted using a scaled-up version of the first MV. This scaling can be proportional to the difference in POC distance.

[0392] In one example, for the second round of DMVR, the list with a longer POC distance can be adjusted using the derived MV, and another list can be adjusted using a scaled-down version of the first MV. This scaling can be proportional to the difference in POC distance.

[0393] o. In one example, for the second round of DMVR, the scaling values ​​in all the above segments can be capped to a predefined minimum and maximum value. This scaling can be exactly the POC distance ratio or its capped version.

[0394] p. In one example, scaling may not be involved for the BDOF DMVR section, and both List 0 and List 1 use the exact same MV adjustment with mirror properties.

[0395] q. In one example, some scaling may be involved for the BDOF DMVR section, and list 0 and list 1 use different MV adjustments.

[0396] r. In one example, scaling may not be involved for the BDOF sample portion, and both List 0 and List 1 use the exact same MV adjustment with mirror properties.

[0397] s. In one example, some scaling may be involved for the BDOF sample portion, and List 0 and List 1 use different final MV adjustments. However, this scaling may or may not be applied during the BDOF formula calculation.

[0398] t. In one example, all the different scaling scenarios explained above for the DMVR section can also be applied to the BDOF DMVR and / or BDOF sample section.

[0399] 26. It is proposed how motion compensation is performed in a DMVR and / or how many times motion compensation is performed, which can depend on the POC distance between at least two reference images.

[0400] a. In one example, there may be at least one new motion compensation calculation for cases with unequal POC distances.

[0401] b. In one example, for the bilateral matching cost calculation between pred0 and pred1, pred0 and pred1 can be derived using the same MV adjustment (and mirror property) for both predictions, regardless of the POC distance difference.

[0402] c. In one example, for the bilateral matching cost calculation between pred0 and pred1, pred0 and pred1 can be derived using different (scaled) MV adjustments.

[0403] d. In one example, the scaling MV adjustment can be rounded to the nearest available prediction (integer pixel or half-pixel level), and the prediction for that point can be used.

[0404] e. In one example, the prediction for the MV adjustment for this scaling can be derived using bilinear interpolation between the closest available predictions (integer pixels or half-pixel levels).

[0405] f. In one example, the actual accurate prediction of the scaling MV adjustment can be derived.

[0406] 27. It is proposed that candidates with non-equivalent POC distances can be added to the one-sided DMVR candidate list.

[0407] a. In one example, up to N candidates with unequal POC distances can be added to the one-sided DMVR candidate list. N can be any positive integer, such as 1, 2, 5, ...

[0408] b. Alternatively, candidates with unequal POC distances may not be added to the one-sided DMVR candidate list.

[0409] Regarding the application of regularization in BDOF DMVR or BDOF sample formulas 28. It is proposed that regularization can be added to the BDOF DMVR / sample formula.

[0410]

[0411] a. In one example, regularization terms r1, r2, r3, r4, r5, and r6 can be added to a BDOF formula and changed to: (s1+r1) vx + (s² + r²) vy = s3+r3 (s2+r4) vx + (s5+r5) vy = s6+r6 r1, r2, r3, r4, r5, and r6 can be any integer or real number, such as -11, 0, 4, 7, 1<<10, 1<<14, ... They can be equal or different from each other.

[0412] b. In one example, r1 = r5, and r2, r3, r4, and r6 are zero. r1 can be any integer.

[0413] c. In one example, only r3 and r6 are non-zero numbers.

[0414] d. In one example, r1, r2, r3, r4, r5, and r6 are fixed numbers.

[0415] e. In another example, r1, r2, r3, r4, r5, r6 are the number of variables and may depend on qp, block size, iteration stage, or any other condition discussed earlier in this patent.

[0416] General aspects 29. In one example, the division operation disclosed in this document may be replaced by a non-division operation, which may share the same or similar logic as the division replacement logic in CCLM or CCCM.

[0417] 30. Whether and / or how the methods described above are applied may depend on the encoded / decoded information.

[0418] a. In one example, the encoded / decoded information may include block size and / or temporal layer and / or strip / image type, color components, etc.

[0419] 31. Whether and / or how the methods described above are applied can be indicated in the bitstream.

[0420] a. Indicators for enabling / disabling or indicating which method to apply can be transmitted via signaling at the sequence level / picture group level / picture level / strip level / piece group level, such as in the sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / strip header / piece group header.

[0421] b. Indicators for enabling / disabling or indicating which method to apply can be transmitted via signals at PB / TB / CB / PU / TU / CU / VPDU / CTU / CTU lines / strips / films / sub-pictures / other types of areas containing more than one sample or pixel.

[0422] Further details of embodiments of this disclosure relating to bidirectional optical flow (BDOF) processes will now be described. The embodiments of this disclosure should be considered as examples illustrating general concepts and should not be interpreted in a narrow sense. Furthermore, these embodiments may be applied individually or in any combination.

[0423] As used herein, the term "block" can refer to color components, sub-pictures, pictures, stripes, slices, codec tree units (CTUs), CTU rows, CTU groups, codec units (CUs), prediction units (PUs), transform units (TUs), codec tree blocks (CTBs), codec blocks (CBs), prediction blocks (PBs), transform blocks (TBs), sub-blocks of video blocks, sub-regions within video blocks, and video processing units comprising multiple samples / pixels, etc. Blocks can be rectangular or non-rectangular.

[0424] Figure 12A flowchart of a method 1200 for video processing according to some embodiments of the present disclosure is shown. Method 1200 can be implemented during the conversion between a current video block and a video bitstream. Figure 12 As shown, method 1200 begins at 1202, where a multi-round BDOF process is iteratively applied to the current video block. The multi-round BDOF processes are of the same type.

[0425] In one example embodiment, each round of the multi-round BDOF process can be a BDOF process for motion vector (MV) refinement. For example, in a BDOF process for MV refinement, at least one offset can be determined to refine the MV of the current video block or a sub-block of the current video block. Alternatively, each round of the multi-round BDOF process can be a BDOF process for sample adjustment, also known as sample-based BDOF. In a BDOF process for sample adjustment, at least one offset can be determined to adjust one or more predicted samples in the current video block or a sub-block of the current video block. It should be understood that the multi-round BDOF process can also be any other suitable type. The scope of this disclosure is not limited in this respect.

[0426] In some embodiments, the current round of the BDOF process can be applied based on the results of the previous round of the BDOF process. Furthermore, the next round of the BDOF process can be applied based on the results of the current round of the BDOF process.

[0427] At 1204, the conversion is performed based on the result of applying a multi-round BDOF process. In some embodiments, the conversion may include encoding the current video block into a bitstream. Alternatively or additionally, the conversion may include decoding the current video block from the bitstream. It should be understood that the above description is for illustrative purposes only. The scope of this disclosure is not limited in this respect.

[0428] Given the above, the same type of BDOF process can be performed iteratively for the current video block. Compared to traditional solutions that do not perform the same type of BDOF process iteratively, the proposed method can advantageously refine the sample values ​​or MV of the current video block more effectively. Therefore, the encoding and decoding quality can be improved.

[0429] In some embodiments, the number of multi-round BDOF processes is equal to N, and N is a non-negative integer, such as 0, 1, 2, 3, 5, 8, etc.

[0430] In some embodiments, the multi-round BDOF process includes a first round of BDOF and a second round of BDOF following the first round. Furthermore, the second round of BDOF is applied based on the results of the first round of BDOF.

[0431] In some embodiments, the number of multiple rounds of BDOF processes may depend on at least one of the following: the size of the current video block, the size of the sub-blocks of the current video block, the quantization parameter (QP) associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of multiple reference video blocks of the current video block, the similarity of multiple predictions from multiple reference video blocks, the distribution of differences between multiple predictions from multiple reference video blocks, the temporal gradients of multiple reference video blocks, the spatial gradients of multiple reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one prediction block of the current video block.

[0432] In one example embodiment, the encoded / decoded information may include residual information, encoding / decoding tools applied to the current video block, etc. Additionally or alternatively, at least one prediction block may include multiple prediction blocks from multiple lists of reference images (such as list 0, list 1, etc.) of the current video block. Furthermore, information about at least one prediction block may include characteristics of at least one prediction block, the size of at least one prediction block, etc. Moreover, quantization parameters associated with the current video block may include quantization parameters of the current video block, quantization parameters of the current codec unit (CU) including the current video block, quantization parameters of the current stripe including the current video block, quantization parameters of the sequence including the current video block, etc.

[0433] In some embodiments, whether to apply the next round of BDOF depends on the result of the current round of BDOF. For example, if the refinement of the MV or sample values ​​determined based on the result of the current round of BDOF is less than a threshold, then the next round of BDOF is not applied.

[0434] Alternatively, the first refinement of MV or sample values ​​is determined based on the results of the current round of BDOF, and the second refinement of MV or sample values ​​is determined based on the results of the previous round of BDOF. If the ratio between the first and second refinements is less than a threshold, the next round of BDOF is not applied.

[0435] In some alternative embodiments, whether to apply the next round of the BDOF process is independent of the result of the current round of the BDOF process.

[0436] In some embodiments, the application of a multi-round BDOF process is terminated after a predetermined number of rounds of the BDOF process has been executed. Alternatively, the application of a multi-round BDOF process is terminated if certain conditions are met. For example, conditions may include a cost metric between two predictions from two lists of reference images for the current video block being less than a threshold, where the two predictions are determined based on MV, which is refined based on the results of the current round of the BDOF process. By way of example and not limitation, cost metrics may include: Sum of Absolute Differences (SAD), Sum of Absolute Transform Differences (SATD), Sum of Squared Differences (SSD), Sum of Absolute Differences with Mean Removal (MR-SAD), etc.

[0437] In some embodiments, the threshold used to determine whether to apply a round of the BDOF process is fixed. Alternatively, the threshold may be different for different rounds of the BDOF process.

[0438] In some embodiments, whether to apply a round of BDOF is determined by comparing a threshold with a cost metric between two predictions from two lists of reference images for the current video block.

[0439] In some embodiments, the number of rounds of BDOF processes is indicated at one of the following: sequence level, picture level, or stripe head.

[0440] In some embodiments, the multi-round BDOF process is applied based on the same sub-block size. For example, the same sub-block size is fixed.

[0441] In some embodiments, different rounds of the BDOF process are applied based on different sub-block sizes. Additionally or alternatively, one round of a multi-round BDOF process is applied based on a different sub-block size. For example, the sub-block size can be adaptive for each iteration or within a single iteration.

[0442] In some embodiments, the sub-block size for applying a round of BDOF processing depends on at least one of the following: the size of the current video block, the QP associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of multiple reference video blocks of the current video block, the similarity of multiple predictions from multiple reference video blocks, the distribution of differences between multiple predictions from multiple reference video blocks, the temporal gradients of multiple reference video blocks, the spatial gradients of multiple reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one prediction block of the current video block.

[0443] In some embodiments, the sub-block size for applying a round of the BDOF process is W×H, and each term in W and H is a non-negative integer, such as 2, 4, 8, 11, 16, etc. For example, iteration j can have a sub-block size W_j. H_j, where j can be any non-negative integer, and W_j and H_j can be any positive integer.

[0444] In some embodiments, the size of the sub-block used to apply the next round of the BDOF process depends on the MV determined based on the result of applying the current round of the BDOF process.

[0445] In some embodiments, multiple rounds of BDOF procedures are applied based on the same scaling factor used to scale the MV offset determined by applying one round of BDOF procedures. For example, the same scaling factor is fixed.

[0446] In some embodiments, the BDOF process of different rounds is applied based on different scaling factors, which are used to scale the MV offset determined by applying the BDOF process of different rounds.

[0447] In some embodiments, the scaling factor used to scale the MV offset determined by applying a round of BDOF process depends on at least one of the following: the size of the current video block, the QP associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of multiple reference video blocks of the current video block, the similarity of multiple predictions from multiple reference video blocks, the distribution of differences between multiple predictions from multiple reference video blocks, the temporal gradients of multiple reference video blocks, the spatial gradients of multiple reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one prediction block of the current video block.

[0448] In some embodiments, the scaling factor used to scale the MV offset determined by applying one round of the BDOF process is equal to a real number, such as 0.25, 0.4, 0.5, 1, 1.63, 2, 4, etc. For example, each iteration (i.e., each round) can have its own scaling factor; for example, iteration j can have a scaling factor s_j.

[0449] In some embodiments, the scaling factor used to scale the MV offset determined by applying the current round of the BDOF process depends on at least one MV determined based on the results of applying the current round of the BDOF process and / or based on the results of a previous round of the BDOF process. For example, the scaling factor depends on at least one of the following: the angle between at least one MV or the magnitude of at least one MV, the QP associated with the current video block, the sub-block size of the current video block, or the size of the current video block.

[0450] In some embodiments, the scaling factor used to scale the sample values ​​determined based on the results of applying a round of BDOF process depends on at least one of the following: the number of rounds of BDOF process, the MV offset determined by applying a round of BDOF process, the size of the current video block, the size of the sub-blocks of the current video block, the quantization parameter (QP) associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of multiple reference video blocks of the current video block, the similarity of multiple predictions from multiple reference video blocks, the distribution of differences between multiple predictions from multiple reference video blocks, the temporal gradients of multiple reference video blocks, the spatial gradients of multiple reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one prediction block of the current video block.

[0451] In some embodiments, the scaling factor is equal to a real number. For example, the scaling factor for a sample-based BDOF procedure is equal to a real number, such as 0.2, 0.25, 0.5, 1.1, etc.

[0452] In some embodiments, during a round of BDOF, at least one MV offset (or simply offset) for refining the MV is determined by adding one or more regularization terms to a set of equations for that round of BDOF. For example, at least one offset is determined based on the following formula:

[0453] Where Gx represents the sum of the horizontal gradient values ​​determined for each of the multiple reference video blocks in the current video block, Gy represents the sum of the vertical gradient values ​​determined for each of the multiple reference video blocks, and dI represents the difference in sample values ​​between the multiple reference video blocks. This represents a weighted sum or summation within the target region for this round of BDOF process. vx represents the first offset in at least one offset, vy represents the second offset in at least one offset, r1 represents the first regularization term in one or more regularization terms, r2 represents the second regularization term in one or more regularization terms, r3 represents the third regularization term in one or more regularization terms, r4 represents the fourth regularization term in one or more regularization terms, r5 represents the fifth regularization term in one or more regularization terms, and r6 represents the sixth regularization term in one or more regularization terms.

[0454] In some embodiments, each of the first, second, third, fourth, fifth, and sixth regularization terms is an integer or a real number, such as -11, 0, 4, 7, 1<<10, 1<<14, etc.

[0455] In some embodiments, the first, second, third, fourth, fifth, and sixth regularization terms are the same. Alternatively, the first, second, third, fourth, fifth, and sixth regularization terms are different from each other.

[0456] In some embodiments, the first regularization term is equal to the fifth regularization term. Additionally, each of the second, third, fourth, and sixth regularization terms is equal to zero.

[0457] In some embodiments, each of the third and sixth regularization terms is not equal to zero. Additionally, each of the first, second, fourth, and fifth regularization terms is equal to zero.

[0458] In some embodiments, each of the first, second, third, fourth, fifth, and sixth regularization terms is fixed.

[0459] In some embodiments, at least one of the first, second, third, fourth, fifth, and sixth regularization terms depends on at least one of the following: the iteration stage, the size of the current video block, the size of the sub-blocks of the current video block, the quantization parameter (QP) associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of multiple reference video blocks of the current video block, the similarity of multiple predictions from multiple reference video blocks, the distribution of the differences between multiple predictions from multiple reference video blocks, the temporal gradients of multiple reference video blocks, the spatial gradients of multiple reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one prediction block of the current video block.

[0460] In view of the above, the solutions according to some embodiments of this disclosure can advantageously improve encoding / decoding efficiency and encoding / decoding quality.

[0461] According to another embodiment of this disclosure, a non-transitory computer-readable recording medium is provided. This non-transitory computer-readable recording medium stores a bitstream of video generated by a method performed by a video processing apparatus. In this method, a multi-round bidirectional optical flow (BDOF) process is iteratively applied to the current video block. The multi-round BDOF processes are of the same type. Furthermore, the bitstream is generated based on the result of applying the multi-round BDOF process.

[0462] According to further embodiments of this disclosure, a method for storing a bitstream of video is provided. In this method, a multi-round bidirectional optical flow (BDOF) process is iteratively applied to the current video block of the video. The multi-round BDOF processes are of the same type. Furthermore, the bitstream is generated based on the results of applying the multi-round BDOF process and is stored in a non-transitory computer-readable recording medium.

[0463] The embodiments of this disclosure can be described according to the following entries, and their features can be combined in any reasonable manner.

[0464] Item 1. A method for video processing, comprising: for a conversion between a current video block and a bitstream of the video, iteratively applying a multi-round bidirectional optical flow (BDOF) process to the current video block, the multi-round BDOF processes being of the same type; and performing the conversion based on the result of applying the multi-round BDOF process.

[0465] Item 2. According to the method of Item 1, each round of the multi-round BDOF process is a BDOF process for motion vector (MV) refinement, or each round of the multi-round BDOF process is a BDOF process for sample adjustment.

[0466] Item 3. The method according to any one of items 1 to 2, wherein the number of said multi-round BDOF processes is equal to N, and N is a non-negative integer.

[0467] Item 4. The method according to any one of Items 1 to 3, wherein the multi-round BDOF process includes a first round BDOF process and a second round BDOF process following the first round BDOF process, and the second round BDOF process is applied based on the result of applying the first round BDOF process.

[0468] Item 5. The method according to any one of Items 1 to 5, wherein the number of said multi-round BDOF processes depends on at least one of the following: the size of the current video block, the size of the sub-blocks of the current video block, the quantization parameter (QP) associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of a plurality of reference video blocks of the current video block, the similarity of a plurality of predictions from the plurality of reference video blocks, the distribution of the differences between the plurality of predictions from the plurality of reference video blocks, the temporal gradient of the plurality of reference video blocks, the spatial gradient of the plurality of reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one prediction block of the current video block.

[0469] Item 6. The method according to any one of items 1 to 5, wherein whether to apply the next round of BDOF process depends on the result of the current round of BDOF process.

[0470] Item 7. The method according to Item 6, wherein if the refinement of the MV or sample value determined based on the result of the current round of BDOF process is less than a threshold, then the next round of BDOF process is not applied.

[0471] Item 8. The method according to Item 6, wherein a first refinement of the MV or sample value is determined based on the result of the current round of BDOF process, a second refinement of the MV or sample value is determined based on the result of a round of BDOF prior to the current round of BDOF process, and the next round of BDOF process is not applied if the ratio between the first refinement and the second refinement is less than a threshold.

[0472] Item 9. The method according to any one of items 1 to 5, wherein whether or not the next round of the BDOF process is applied is independent of the result of the current round of the BDOF process.

[0473] Item 10. The method according to any one of items 1 to 9, wherein the application of the multi-round BDOF process is terminated after a predetermined number of rounds of BDOF process has been executed.

[0474] Item 11. The method according to any one of items 1 to 10, wherein the application of the multi-round BDOF process is terminated if the condition is met.

[0475] Item 12. The method according to Item 11, wherein the condition includes a cost metric between two predictions from two lists of reference images for the current video block being less than a threshold, the two predictions being determined based on MV, the MV being refined based on the result of applying the current round of the BDOF process.

[0476] Item 13. The method according to Item 12, wherein the cost metric includes at least one of the following: sum of absolute differences (SAD), sum of absolute transformation differences (SATD), sum of squared differences (SSD), or sum of absolute differences with mean removed (MR-SAD).

[0477] Item 14. The method according to any one of items 1 to 13, wherein the threshold for determining whether to apply a round of BDOF process is fixed, or the threshold is different for different rounds of BDOF process.

[0478] Item 15. The method according to Item 14, wherein whether to apply a round of BDOF process is determined by comparing the threshold with a cost metric between two predictions from two lists of reference images for the current video block.

[0479] Item 16. The method according to any one of Items 1 to 15, wherein the number of said multi-round BDOF processes is indicated at one of the following: sequence level, picture level, or strip head.

[0480] Item 17. The method according to any one of Items 1 to 16, wherein the multi-round BDOF process is applied based on the same sub-block size.

[0481] Item 18. The method according to Item 17, wherein the same sub-block size is fixed.

[0482] Item 19. The method according to any one of items 1 to 16, wherein the BDOF process of different rounds is applied based on different sub-block sizes, and / or one round of the multi-round BDOF process is applied based on different sub-block sizes.

[0483] Item 20. The method according to any one of Items 1 to 19, wherein the sub-block size for applying a round of BDOF process depends on at least one of the following: the size of the current video block, the QP associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of a plurality of reference video blocks of the current video block, the similarity of a plurality of predictions from the plurality of reference video blocks, the distribution of differences between the plurality of predictions from the plurality of reference video blocks, the temporal gradient of the plurality of reference video blocks, the spatial gradient of the plurality of reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one prediction block of the current video block.

[0484] Item 21. The method according to any one of items 1 to 20, wherein the size of the sub-block used to apply a round of BDOF process is W×H, and each of W and H is a non-negative integer.

[0485] Item 22. The method according to any one of items 1 to 21, wherein the size of the sub-block used to apply the next round of the BDOF process depends on the MV determined based on the result of applying the current round of the BDOF process.

[0486] Item 23. The method according to any one of items 1 to 22, wherein the multi-round BDOF process is applied based on the same scaling factor, which is used to scale the MV offset determined by applying a single round of BDOF process.

[0487] Item 24. The method according to Item 23, wherein the same scaling factor is fixed.

[0488] Item 25. The method according to any one of items 1 to 22, wherein the BDOF process of different rounds is applied based on different scaling factors, the different scaling factors being used to scale the MV offset determined by applying the BDOF process of the different rounds respectively.

[0489] Item 26. The method according to any one of Items 1 to 23, wherein the scaling factor for scaling the MV offset determined by applying a round of BDOF process depends on at least one of the following: the size of the current video block, the QP associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of a plurality of reference video blocks of the current video block, the similarity of a plurality of predictions from the plurality of reference video blocks, the distribution of differences between the plurality of predictions from the plurality of reference video blocks, the temporal gradient of the plurality of reference video blocks, the spatial gradient of the plurality of reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one prediction block of the current video block.

[0490] Item 27. The method according to any one of items 1 to 26, wherein the scaling factor used to scale the MV offset determined by applying a round of BDOF procedure is equal to a real number.

[0491] Item 28. The method according to any one of Items 1 to 27, wherein the scaling factor for scaling the MV offset determined by applying the current round of BDOF process depends on at least one MV determined based on at least one of the following: the result of applying the current round of BDOF process, or the result of applying a round of BDOF process prior to the current round of BDOF process.

[0492] Item 29. The method according to Item 28, wherein the scaling factor depends on at least one of the following: the angle between the at least one MV, or the amplitude of the at least one MV, the QP associated with the current video block, the sub-block size of the current video block, or the size of the current video block.

[0493] Item 30. According to any one of items 1 to 29, the scaling factor used to scale the sample values ​​determined based on the result of applying a round of BDOF process depends on at least one of the following: the number of rounds of BDOF process, the MV offset determined by applying the round of BDOF process, the size of the current video block, the size of the sub-blocks of the current video block, the quantization parameter (QP) associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of a plurality of reference video blocks of the current video block, the similarity of a plurality of prediction values ​​from the plurality of reference video blocks, the distribution of differences between a plurality of prediction values ​​from the plurality of reference video blocks, the temporal gradient of the plurality of reference video blocks, the spatial gradient of the plurality of reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one prediction block of the current video block.

[0494] Item 31. The method according to Item 30, wherein the scaling factor is equal to a real number.

[0495] Item 32. The method according to any one of items 1 to 31, wherein in a round of BDOF process, at least one offset for refining MV is determined with one or more regularization terms added to a set of equations for said round of BDOF process.

[0496] Item 33. According to the method described in Item 32, wherein the at least one offset is determined based on the following formula

[0497] Where Gx represents the summation of the horizontal gradient values ​​determined for each of the multiple reference video blocks of the current video block, Gy represents the summation of the vertical gradient values ​​determined for each of the multiple reference video blocks, and dI represents the difference in sample values ​​between the multiple reference video blocks. This represents a weighted sum or summation within the target region for the first round of the BDOF process; vx represents the first offset among the at least one offset; vy represents the second offset among the at least one offset; r1 represents the first regularization term among the one or more regularization terms; r2 represents the second regularization term among the one or more regularization terms; r3 represents the third regularization term among the one or more regularization terms; r4 represents the fourth regularization term among the one or more regularization terms; r5 represents the fifth regularization term among the one or more regularization terms; and r6 represents the sixth regularization term among the one or more regularization terms.

[0498] Item 34. The method according to Item 33, wherein each of the first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term is an integer or a real number.

[0499] Item 35. The method according to any one of items 33 to 34, wherein the first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term are the same, or the first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term are different from each other.

[0500] Item 36. The method according to any one of items 33 to 35, wherein the first regularization term is equal to the fifth regularization term, and each of the second, third, fourth, and sixth regularization terms is equal to zero.

[0501] Item 37. The method according to any one of items 33 to 35, wherein each of the third regularization term and the sixth regularization term is not equal to zero, and each of the first regularization term, the second regularization term, the fourth regularization term, and the fifth regularization term is equal to zero.

[0502] Item 38. The method according to any one of items 33 to 37, wherein each of the first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term is fixed.

[0503] Item 39. The method according to any one of items 33 to 37, wherein at least one of the first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term depends on at least one of the following: the iteration phase, the size of the current video block, the size of the sub-blocks of the current video block, the quantization parameter (QP) associated with the current video block, the temporal layer of the current video block, the sequence resolution associated with the current video block, the characteristics of a plurality of reference video blocks of the current video block, and the characteristics from the plurality of reference... The similarity of multiple predicted values ​​of a video block, the distribution of differences between multiple predicted values ​​of the multiple reference video blocks, the temporal gradient of the multiple reference video blocks, the spatial gradient of the multiple reference video blocks, the prediction type, the adjustment value determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR), the adjustment value determined in the second pass of the multi-pass DMVR, the sequence resolution, the color components of the current video block, the color format of the current video block, the encoded / decoded information of the current video block, or the information of at least one predicted block of the current video block.

[0504] Item 40. The method according to any one of items 1 to 39, wherein the conversion includes encoding the current video block into the bitstream.

[0505] Item 41. The method according to any one of items 1 to 39, wherein the conversion includes decoding the current video block from the bitstream.

[0506] Item 42. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of items 1 to 41.

[0507] Item 43. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method according to any one of items 1 to 41.

[0508] Item 44. A non-transitory computer-readable recording medium for storing a bitstream of video, the bitstream of video being generated by a method performed by means of a video processing apparatus, wherein the method comprises: iteratively applying a multi-round bidirectional optical flow (BDOF) process to a current video block of the video, the multi-round BDOF processes being of the same type; and generating the bitstream based on the result of applying the multi-round BDOF process.

[0509] Item 45. A method for storing a bitstream of video, comprising: iteratively applying a multi-round bidirectional optical flow (BDOF) process to a current video block of the video, the multi-round BDOF processes being of the same type; generating the bitstream based on the result of applying the multi-round BDOF process; and storing the bitstream in a non-transitory computer-readable recording medium.

[0510] Example device Figure 13 A block diagram of a computing device 1300 in which various embodiments of the present disclosure may be implemented is shown. The computing device 1300 may be implemented as a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300), or may be included in a source device 110 (or video encoder 114 or 200) or a destination device 120 (or video decoder 124 or 300).

[0511] It should be understood that, Figure 13 The computing device 1300 shown is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of this disclosure.

[0512] like Figure 13 As shown, computing device 1300 includes general-purpose computing device 1300. Computing device 1300 may include at least one or more processors or processing units 1310, memory 1320, storage unit 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360.

[0513] In some embodiments, the computing device 1300 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, large computing device, etc., provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia tablet computers, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, and includes accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 1300 can support any type of interface to the user (such as "wearable" circuitry devices, etc.).

[0514] Processing unit 1310 can be a physical processor or a virtual processor, and can perform various processes based on programs stored in memory 1320. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of computing device 1300. Processing unit 1310 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0515] Computing device 1300 typically includes various computer storage media. Such media can be any media accessible by computing device 1300, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 1320 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage cell 1330 can be any removable or non-removable media and may include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or data and can be accessed within computing device 1300.

[0516] The computing device 1300 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Although in Figure 13 Not shown, but a disk drive for reading from and / or writing to a removable non-volatile disk, and an optical disc drive for reading from and / or writing to a removable non-volatile optical disc may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0517] Communication unit 1340 communicates with another computing device via a communication medium. Additionally, the functionality of components in computing device 1300 can be implemented by a single computing cluster or multiple computing machines that can communicate via communication connections. Therefore, computing device 1300 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0518] Input device 1350 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 1360 can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 1340, computing device 1300 can also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 1300 can also communicate with one or more devices that enable a user to interact with computing device 1300, or, if needed, with any device (e.g., network card, modem, etc.) that enables computing device 1300 to communicate with one or more other computing devices. Such communication can be performed via an input / output (I / O) interface (not shown).

[0519] In some embodiments, some or all of the components of computing device 1300 may be arranged in a cloud computing architecture, rather than integrated into a single device. In a cloud computing architecture, components may be remotely provided and work together to achieve the functionality described herein. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services via a wide area network (WAN), such as the Internet, using suitable protocols. For example, a cloud computing provider provides applications via a WAN that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed across remote data center locations. Cloud computing infrastructure may provide services through shared data centers, although they appear as a single access point to users. Therefore, a cloud computing architecture can be used to provide the components and functionality described herein from a service provider at a remote location. Alternatively, the components and functionality described herein may be provided by conventional servers or installed directly or otherwise on client devices.

[0520] In embodiments of this disclosure, computing device 1300 may be used to implement video encoding / decoding. Memory 1320 may include one or more video codec modules 1325 having one or more program instructions. These modules are accessible and executable by processing unit 1310 to perform the functions of the various embodiments described herein.

[0521] In an example embodiment of performing video encoding, input device 1350 may receive video data as input 1370 to be encoded. The video data may be processed, for example, by video codec module 1325 to generate an encoded bitstream. The encoded bitstream may be provided as output 1380 via output device 1360.

[0522] In an example embodiment of performing video decoding, input device 1350 may receive an encoded bitstream as input 1370. The encoded bitstream may be processed, for example, by video codec module 1325 to generate decoded video data. The decoded video data may be provided as output 1380 via output device 1360.

[0523] While this disclosure has been specifically shown and described with reference to preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application as defined by the appended claims. These variations are intended to be covered by the scope of this application. Therefore, the foregoing description of embodiments of this application is not intended to be limiting.

Claims

1. A method for video processing, comprising: For the conversion between the current video block and the bitstream of the video, a multi-round bidirectional optical flow (BDOF) process is iteratively applied to the current video block, wherein the multi-round BDOF processes are of the same type; and The transformation is performed based on the results of the multi-round BDOF process described above.

2. The method of claim 1, wherein each round of the multi-round BDOF process is a BDOF process for motion vector (MV) refinement, or Each round of the multi-round BDOF process is a BDOF process used for sample point adjustment.

3. The method according to any one of claims 1 to 2, wherein the number of said multi-round BDOF processes is equal to N, and N is a non-negative integer.

4. The method according to any one of claims 1 to 3, wherein the multi-round BDOF process includes a first round BDOF process and a second round BDOF process following the first round BDOF process, and the second round BDOF process is applied based on the result of applying the first round BDOF process.

5. The method according to any one of claims 1 to 5, wherein the number of said multi-round BDOF processes depends on at least one of the following: The size of the current video block The size of the sub-blocks of the current video block, Quantization parameters (QP) associated with the current video block. The temporal layer of the current video block, The sequence resolution associated with the current video block, The characteristics of the multiple reference video blocks of the current video block, The similarity of multiple predicted values ​​from the plurality of reference video blocks, The distribution of differences between multiple predicted values ​​from the plurality of reference video blocks. The temporal gradients of the plurality of reference video blocks, The spatial gradient of the plurality of reference video blocks, Prediction type The adjustment values ​​determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR) are... The adjustment value determined in the second pass of the multiple passes of DMVR Sequence resolution, The color components of the current video block, The color format of the current video block, The encoded and decoded information of the current video block, or Information about at least one predicted block of the current video block.

6. The method according to any one of claims 1 to 5, wherein whether to apply the next round of BDOF process depends on the result of the current round of BDOF process.

7. The method of claim 6, wherein if the refinement of the MV or sample value determined based on the result of the current round of BDOF process is less than a threshold, then the next round of BDOF process is not applied.

8. The method of claim 6, wherein a first refinement of the MV or sample value is determined based on the result of the current round of BDOF process, a second refinement of the MV or sample value is determined based on the result of a round of BDOF prior to the current round of BDOF process, and the next round of BDOF process is not applied if the ratio between the first refinement and the second refinement is less than a threshold.

9. The method according to any one of claims 1 to 5, wherein whether or not the next round of the BDOF process is applied is independent of the result of the current round of the BDOF process.

10. The method according to any one of claims 1 to 9, wherein the application of the multi-round BDOF process is terminated after a predetermined number of rounds of BDOF process have been executed.

11. The method according to any one of claims 1 to 10, wherein the application of the multi-round BDOF process is terminated if the condition is met.

12. The method of claim 11, wherein the condition includes a cost metric between two predictions from two lists of reference images for the current video block being less than a threshold, the two predictions being determined based on MV, the MV being refined based on the result of applying the current round of the BDOF process.

13. The method of claim 12, wherein the cost metric comprises at least one of the following: Sum of absolute differences (SAD) Sum of Absolute Transform Differences (SATD) Sum of Squared Differences (SSD), or The sum of absolute differences after removing the mean (MR-SAD).

14. The method according to any one of claims 1 to 13, wherein the threshold for determining whether to apply a round of BDOF process is fixed, or The threshold is different for different rounds of the BDOF process.

15. The method of claim 14, wherein whether to apply a round of BDOF process is determined by comparing the threshold with a cost metric between two predictions from two lists of reference images for the current video block.

16. The method according to any one of claims 1 to 15, wherein the number of said multi-round BDOF processes is indicated in one of the following: sequence level, Image level, or Strip head.

17. The method according to any one of claims 1 to 16, wherein the multi-round BDOF process is applied based on the same sub-block size.

18. The method of claim 17, wherein the size of the identical sub-blocks is fixed.

19. The method according to any one of claims 1 to 16, wherein the BDOF process for different wheels is applied based on different sub-block sizes, and / or In the multi-round BDOF process, one round is applied based on a different sub-block size.

20. The method according to any one of claims 1 to 19, wherein the size of the sub-block used to apply a round of BDOF process depends on at least one of the following: The size of the current video block The QP associated with the current video block, The temporal layer of the current video block, The sequence resolution associated with the current video block, The characteristics of the multiple reference video blocks of the current video block, The similarity of multiple predicted values ​​from the plurality of reference video blocks, The distribution of differences between multiple predicted values ​​from the plurality of reference video blocks. The temporal gradients of the plurality of reference video blocks, The spatial gradient of the plurality of reference video blocks, Prediction type The adjustment values ​​determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR) are... The adjustment value determined in the second pass of the multiple passes of DMVR Sequence resolution, The color components of the current video block, The color format of the current video block, The encoded and decoded information of the current video block, or Information about at least one predicted block of the current video block.

21. The method according to any one of claims 1 to 20, wherein the size of the sub-block used to apply a round of BDOF process is W×H, and each of W and H is a non-negative integer.

22. The method according to any one of claims 1 to 21, wherein the sub-block size for applying the next round of the BDOF process depends on the MV determined based on the result of applying the current round of the BDOF process.

23. The method according to any one of claims 1 to 22, wherein the multi-round BDOF process is applied based on the same scaling factor, the same scaling factor being used to scale the MV offset determined by applying a single round of BDOF process.

24. The method of claim 23, wherein the same scaling factor is fixed.

25. The method according to any one of claims 1 to 22, wherein the BDOF process of different rounds is applied based on different scaling factors, the different scaling factors being used to scale the MV offset determined by applying the BDOF process of the different rounds.

26. The method according to any one of claims 1 to 23, wherein the scaling factor used to scale the MV offset determined by applying a round of BDOF process depends on at least one of the following: The size of the current video block The QP associated with the current video block, The temporal layer of the current video block, The sequence resolution associated with the current video block, The characteristics of the multiple reference video blocks of the current video block, The similarity of multiple predicted values ​​from the plurality of reference video blocks, The distribution of differences between multiple predicted values ​​from the plurality of reference video blocks. The temporal gradients of the plurality of reference video blocks, The spatial gradient of the plurality of reference video blocks, Prediction type The adjustment values ​​determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR) are... The adjustment value determined in the second pass of the multiple passes of DMVR Sequence resolution, The color components of the current video block, The color format of the current video block, The encoded and decoded information of the current video block, or Information about at least one predicted block of the current video block.

27. The method according to any one of claims 1 to 26, wherein the scaling factor used to scale the MV offset determined by applying a round of BDOF process is equal to a real number.

28. The method according to any one of claims 1 to 27, wherein the scaling factor used to scale the MV offset determined by applying the current round of the BDOF process depends on at least one MV determined based on at least one of the following: Apply the results of the current round of the BDOF process, or The result of the previous BDOF process is applied to the current BDOF process.

29. The method of claim 28, wherein the scaling factor depends on at least one of the following: The angle between at least one MV, or The amplitude of at least one MV, The QP associated with the current video block, The size of the sub-blocks of the current video block, or The size of the current video block.

30. The method according to any one of claims 1 to 29, wherein the scaling factor used for scaling the sample values ​​determined based on the results of applying one round of BDOF process depends on at least one of the following: The number of the multi-round BDOF process, The MV offset is determined by applying the aforementioned BDOF process. The size of the current video block The size of the sub-blocks of the current video block, Quantization parameters (QP) associated with the current video block. The temporal layer of the current video block, The sequence resolution associated with the current video block, The characteristics of the multiple reference video blocks of the current video block, The similarity of multiple predicted values ​​from the plurality of reference video blocks, The distribution of differences between multiple predicted values ​​from the plurality of reference video blocks. The temporal gradients of the plurality of reference video blocks, The spatial gradient of the plurality of reference video blocks, Prediction type The adjustment values ​​determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR) are... The adjustment value determined in the second pass of the multiple passes of DMVR Sequence resolution, The color components of the current video block, The color format of the current video block, The encoded and decoded information of the current video block, or Information about at least one predicted block of the current video block.

31. The method of claim 30, wherein the scaling factor is equal to a real number.

32. The method according to any one of claims 1 to 31, wherein in a round of BDOF process, at least one offset for refining MV is determined with one or more regularization terms added to a set of equations for said round of BDOF process.

33. The method of claim 32, wherein the at least one offset is determined based on the following formula: Where Gx represents the summation of the horizontal gradient values ​​determined for each of the multiple reference video blocks of the current video block, Gy represents the summation of the vertical gradient values ​​determined for each of the multiple reference video blocks, and dI represents the difference in sample values ​​between the multiple reference video blocks. This represents a weighted sum or summation within the target region for the first round of the BDOF process; vx represents the first offset among the at least one offset; vy represents the second offset among the at least one offset; r1 represents the first regularization term among the one or more regularization terms; r2 represents the second regularization term among the one or more regularization terms; r3 represents the third regularization term among the one or more regularization terms; r4 represents the fourth regularization term among the one or more regularization terms; r5 represents the fifth regularization term among the one or more regularization terms; and r6 represents the sixth regularization term among the one or more regularization terms.

34. The method of claim 33, wherein each of the first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term is an integer or a real number.

35. The method according to any one of claims 33 to 34, wherein the first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term are the same, or The first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term are different from each other.

36. The method according to any one of claims 33 to 35, wherein the first regularization term is equal to the fifth regularization term, and each of the second regularization term, the third regularization term, the fourth regularization term, and the sixth regularization term is equal to zero.

37. The method according to any one of claims 33 to 35, wherein each of the third regularization term and the sixth regularization term is not equal to zero, and each of the first regularization term, the second regularization term, the fourth regularization term, and the fifth regularization term is equal to zero.

38. The method according to any one of claims 33 to 37, wherein each of the first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term is fixed.

39. The method according to any one of claims 33 to 37, wherein at least one of the first regularization term, the second regularization term, the third regularization term, the fourth regularization term, the fifth regularization term, and the sixth regularization term depends on at least one of the following: Iteration phase The size of the current video block The size of the sub-blocks of the current video block, Quantization parameters (QP) associated with the current video block. The temporal layer of the current video block, The sequence resolution associated with the current video block, The characteristics of the multiple reference video blocks of the current video block, The similarity of multiple predicted values ​​from the plurality of reference video blocks, The distribution of differences between multiple predicted values ​​from the plurality of reference video blocks. The temporal gradients of the plurality of reference video blocks, The spatial gradient of the plurality of reference video blocks, Prediction type The adjustment values ​​determined in the first pass of the multi-pass decoder-side motion vector refinement (DMVR) are... The adjustment value determined in the second pass of the multiple passes of DMVR Sequence resolution, The color components of the current video block, The color format of the current video block, The encoded and decoded information of the current video block, or Information about at least one predicted block of the current video block.

40. The method of any one of claims 1 to 39, wherein the conversion comprises encoding the current video block into the bitstream.

41. The method according to any one of claims 1 to 39, wherein the conversion comprises decoding the current video block from the bitstream.

42. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 41.

43. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method according to any one of claims 1 to 41.

44. A non-transitory computer-readable recording medium storing a bitstream of video, the bitstream of video being generated by a method performed by means of a video processing apparatus, wherein the method includes: A multi-round bidirectional optical flow (BDOF) process is iteratively applied to the current video block of the video, wherein the multi-round BDOF processes are of the same type; as well as The bitstream is generated based on the results of the multi-round BDOF process described above.

45. A method for storing a bitstream of video, comprising: A multi-round bidirectional optical flow (BDOF) process is iteratively applied to the current video block of the video, wherein the multi-round BDOF processes are of the same type; The bit stream is generated based on the results of the multi-round BDOF process described above; as well as The bitstream is stored in a non-transitory computer-readable recording medium.

Images