Optical flow-based motion improvement

Abstract

Methods and apparatus for optical flow based motion improvement are provided. An exemplary method includes dividing a coding block into a first set of sub-blocks and a second set of sub-blocks; executing a first round of optical flow-based motion vector improvement on the first group of sub-blocks; and executing a second round of optical flow-based motion vector improvement on the second group of sub-blocks.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This disclosure claims the benefit of priority to U.S. Provisional Application No. 63 / 495,768, filed April 12, 2023, U.S. Provisional Application No. 63 / 508,010, filed June 14, 2023, and U.S. Application No. 18 / 628,086, filed April 5, 2024, the contents of all of which are incorporated herein by reference in their entirety. Technical Field

[0002] The present disclosure relates generally to video processing, and more particularly, to methods and apparatus for optical flow-based motion refinement. Background Art

[0003] A video is composed of a set of static images (or "frames") that capture visual information. In order to reduce storage memory and transmission bandwidth, the video can be compressed before storage or transmission, and decompressed before display. The compression process is usually called encoding, and the decompression process is usually called decoding. There are many video coding formats that use standardized video coding techniques, the most common of which are based on prediction, transform, quantization, entropy coding, and loop filtering. Standardization organizations have developed video coding standards that specify specific video coding formats, such as the High Efficiency Video Coding (HEVC / H.265) standard, the Versatile Video Coding (VVC / H.266) standard, and the AVS standard. As more and more advanced video coding technologies are incorporated into video standards, the coding efficiency of new video coding standards is also getting higher and higher. Summary of the Invention

[0004] Embodiments of the present disclosure provide a method and apparatus for motion improvement based on optical flow.

[0005] According to some exemplary embodiments, a video processing method is provided, including: dividing a coding block into a first group of sub-blocks and a second group of sub-blocks; performing a first round of optical flow-based motion vector improvement on the first group of sub-blocks; and performing a second round of optical flow-based motion vector improvement on the second group of sub-blocks.

[0006] According to some exemplary embodiments, a device is provided, comprising: a memory storing computer instructions; and one or more processors configured to execute the computer instructions. Execution of the computer instructions causes the device to perform operations comprising: dividing a coding block into a first group of sub-blocks and a second group of sub-blocks; performing a first round of optical flow-based motion vector refinement on the first group of sub-blocks; and performing a second round of optical flow-based motion vector refinement on the second group of sub-blocks.

[0007] According to some exemplary embodiments, a non-transitory computer-readable storage medium storing a bitstream of a video is provided. The bitstream is used for processing according to a method, the method comprising: dividing a coding block into a first group of sub-blocks and a second group of sub-blocks; performing a first round of optical flow-based motion vector refinement on the first group of sub-blocks; and performing a second round of optical flow-based motion vector refinement on the second group of sub-blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0008]

[0011] Embodiments and aspects of the present disclosure are illustrated in the following detailed description and accompanying drawings.The various features shown in the drawings are not drawn to scale.

[0009] Figure 1 is a schematic diagram illustrating the structure of an example video sequence according to some embodiments of the present disclosure.

[0010] Figure 2A is a schematic diagram illustrating an example encoding process of a hybrid video coding system according to some embodiments of the present disclosure.

[0011] Figure 2B is a schematic diagram illustrating another example encoding process of a hybrid video coding system according to some embodiments of the present disclosure.

[0012] Figure 3A is a diagram illustrating an example decoding process of a hybrid video coding system according to some embodiments of the present disclosure.

[0013] Figure 3B is a schematic diagram illustrating another example decoding process of a hybrid video coding system according to some embodiments of the present disclosure.

[0014] Figure 4 is a block diagram of an example apparatus for encoding or decoding video according to some embodiments of the present disclosure.

[0015] Figure 5 is a schematic diagram illustrating an expanded coding unit (CU) region used in bidirectional optical flow (BDOF) according to some embodiments of the present disclosure.

[0016] Figure 6 is a schematic diagram illustrating a decoding-side motion vector refinement (DMVR) process according to some embodiments of the present disclosure.

[0017] Figure 7 is a schematic diagram illustrating a 3×3 square search pattern according to some embodiments of the present disclosure.

[0018] Figure 8 FIG. 2 is a schematic diagram illustrating a diamond-shaped area in a search area for a modified motion vector (MV) according to some embodiments of the present disclosure.

[0019] Figure 9 is a flowchart of an exemplary method for performing motion vector improvement and motion compensation according to some embodiments of the present disclosure.

[0020] Figure 10 is a flowchart of an exemplary method for performing motion vector improvement and motion compensation according to some embodiments of the present disclosure.

[0021] Figure 11 is a flowchart of another exemplary method for performing motion vector improvement and motion compensation according to some embodiments of the present disclosure.

[0022] Figure 12A and 12B is a diagram illustrating a process for splitting a prediction block by using an extended prediction block according to some embodiments of the present disclosure.

[0023] Figure 13A and 13B is a diagram illustrating a process for splitting a prediction block by using an extended prediction block according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] Reference will now be made in detail to the exemplary embodiments, examples of which are shown in the accompanying drawings. The following description refers to the accompanying drawings, in which, unless otherwise indicated, the same numbers in different figures represent the same or similar elements. The embodiments set forth in the following description of the exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with the relevant aspects of the present invention described in the appended claims. Specific aspects of the present disclosure are described in more detail below. In the event of a conflict with a term or definition incorporated by reference, the terms and definitions provided herein shall prevail.

[0025] Embodiments disclosed herein relate to encoding and decoding video information, and more particularly, to methods and systems for performing optical flow-based motion vector refinement. The disclosed method performs multiple rounds of optical flow-based motion vector refinement and adaptively adjusts sub-block matrix sizes to produce more stable results in the optical flow calculation.

[0026] In order to reduce the storage space and transmission bandwidth required for these applications, the video can be compressed before being stored or transmitted and decompressed before being displayed. The compression and decompression can be implemented by software or dedicated hardware executed by a processor (e.g., a processor of a general-purpose computer). The module used for compression is generally referred to as an "encoder", while the module used for decompression is generally referred to as a "decoder". The encoder and decoder can be collectively referred to as "codecs". The encoder and decoder can be implemented as any of various suitable hardware, software, or combinations thereof. For example, the hardware implementation of the encoder and decoder can include circuit systems, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, or any combination thereof. The software implementation of the encoder and decoder can include program code, computer-executable instructions, firmware, or any suitable computer-implemented algorithm or process fixed in a computer-readable medium. Video compression and decompression can be implemented by various algorithms or standards, such as MPEG-1, MPEG-2, MPEG-4, H.26x, AVS series, etc. In some applications, the codec may decompress the video according to a first coding standard and recompress the decompressed video using a second coding standard, in which case the codec may be referred to as a "transcoder."

[0027] Video encoding processes identify and retain information that is useful for reconstructing an image, while ignoring information that is unimportant for reconstruction. If the omitted, unimportant information cannot be fully reconstructed, the encoding process is called "lossy." Otherwise, it is called "lossless." Most encoding processes are lossy as a trade-off to reduce required storage space and transmission bandwidth.

[0028] Useful information about an image being encoded (referred to as the "current image") includes changes relative to a reference image (e.g., a previously encoded and reconstructed image). Such changes can include changes in pixel position, brightness, or color, with position changes being of particular interest. A change in the position of a group of pixels representing an object can reflect the object's motion between the reference image and the current image.

[0029] A picture that is encoded without reference to another picture (i.e., its reference picture is itself) is called an "I picture." A picture is called a "P picture" if some or all blocks in the picture (e.g., these blocks typically refer to portions of a video picture) are predicted using intra-frame prediction or inter-frame prediction with one reference picture (e.g., unidirectional prediction). A picture is called a "B picture" if at least one block in the picture is predicted using two reference pictures (e.g., bidirectional prediction).

[0030] Figure 1 、 2A , 2B, 3A, 3B and 4 show the general aspects of the video encoding / decoding apparatus and process used in the disclosed embodiments. Specifically, Figure 1 The structure of an example video sequence 100 according to some embodiments of the present disclosure is shown. Video sequence 100 can be live video, or video that has been captured and archived. Video 100 can be real-life video, computer-generated video (e.g., computer game video), or a combination thereof (e.g., real-life video with augmented reality effects). Video sequence 100 can be from a video capture device (e.g., a camera), a video archive containing previously captured video (e.g., a video file stored in a storage device), or a video feed interface (e.g., a video broadcast transceiver) that receives video from a video content provider.

[0031] like Figure 1 As shown, video sequence 100 may include a series of images arranged in time along a timeline, including images 102, 104, 106, and 108. Images 102-106 are consecutive, and there are more images between images 106 and 108. Figure 1 , picture 102 is an I picture, and its reference picture is picture 102 itself. Picture 104 is a P picture, and its reference picture is picture 102, as indicated by the arrows. Picture 106 is a B picture, and its reference pictures are pictures 104 and 108, as indicated by the arrows. In some embodiments, the reference picture of a picture (e.g., picture 104) may not be immediately before or after the picture. For example, the reference picture of picture 104 may be the picture before picture 102. It should be noted that the reference pictures of pictures 102-106 are merely examples, and the present disclosure does not limit the embodiments of the reference pictures to Figure 1 Example shown.

[0032] Typically, video codecs do not encode or decode an entire image at once due to the computational complexity of such a task. Instead, they may split the image into multiple basic segments and encode or decode the image segment by segment. Such basic segments are referred to in this disclosure as basic processing units ("BPUs"). For example, Figure 1Structure 110 in shows an example structure of an image (e.g., any of images 102-108) of video sequence 100. In structure 110, the image is divided into 4×4 basic processing units, the boundaries of which are shown as dashed lines. In some embodiments, the basic processing units may be referred to as "macroblocks" in some video coding standards (e.g., the MPEG family, H.261, H.263, or H.264 / AVC), or as "coding tree units" ("CTUs") in some other video coding standards (e.g., H.265 / HEVC, H.266 / VVC, or AVS). The basic processing units may have variable sizes in an image, such as 128×128, 64×64, 32×32, 16×16, 4×8, 16×32, or any arbitrary shape and size of pixels. The size and shape of the basic processing units may be selected for an image based on a balance between coding efficiency and the level of detail to be maintained in the basic processing units.

[0033] The basic processing unit can be a logical unit that can include a set of different types of video data stored in a computer memory (e.g., stored in a video frame buffer). For example, a basic processing unit of a color image can include a luma component (Y) representing non-color brightness information, one or more chroma components (e.g., Cb and Cr) representing color information, and associated syntax elements, wherein the luma and chroma components can have the same size as the basic processing unit. In some video coding standards (e.g., H.265 / HEVC, H.266 / VVC, or AVS), the luma and chroma components can be referred to as "coding tree blocks" ("CTBs"). Any operation performed on a basic processing unit can be repeated for each of the luma and chroma components of the basic processing unit.

[0034] Video coding involves multiple stages of operation, examples of which are shown in Figures 2A-2B and Figures 3A-3B. For each level, the size of the basic processing unit may still be too large to be processed, so it can be further divided into segments referred to as "basic processing sub-units" in this disclosure. In some embodiments, the basic processing sub-unit may be called a "block" in some video coding standards (e.g., MPEG family, H.261, H.263, H.264 / AVC or AVS), or a "coding unit" ("CU") in some other video coding standards (e.g., H.265 / HEVC, H.266 / VVC or AVS). The basic processing sub-unit may have the same size as the basic processing unit or a smaller size than the basic processing unit. Similar to the basic processing unit, the basic processing sub-unit is also a logical unit, which may include a set of different types of video data (e.g., Y, Cb, Cr and associated syntax elements) stored in computer memory (e.g., stored in a video frame buffer). Any operation performed on the basic processing sub-unit can be repeated for each of the luminance and chrominance components of the basic processing sub-unit. It should be noted that this division can be performed to more levels depending on processing needs. It should also be noted that different stages may use different schemes to partition the basic processing units.

[0035] For example, in the mode decision phase (an example of which is Figure 2B ), the encoder can decide what prediction mode (e.g., intra-image prediction or inter-image prediction) to use for a basic processing unit, which may be too large to make such a decision. The encoder can split the basic processing unit into multiple basic processing sub-units (e.g., as CUs in H.265 / HEVC, H.266 / VVC, or AVS) and decide the prediction type for each individual basic processing sub-unit.

[0036] For another example, in the prediction phase (an example of which is Figures 2A-2B As shown in FIG, 1 , the encoder may perform prediction operations at the level of a basic processing sub-unit (e.g., a CU). However, in some cases, the basic processing sub-unit may still be too large to process. The encoder may further split the basic processing sub-unit into smaller segments (e.g., "prediction blocks" or "PBs" in H.265 / HEVC, H.266 / VVC, or AVS), at which the prediction operations may be performed.

[0037] For another example, in the transformation phase (an example of which is Figures 2A-2BAs shown), the encoder may perform a transform operation on the residual basic processing sub-unit (e.g., CU). However, in some cases, the basic processing sub-unit may still be too large to be processed. The encoder may further divide the basic processing sub-unit into smaller segments (e.g., referred to as "transform blocks" or "TBs" in H.265 / HEVC, H.266 / VVC, or AVS), at the level of which the transform operation may be performed. It should be noted that the partitioning schemes for the same basic processing sub-unit may be different in the prediction stage and the transform stage. For example, in H.265 / HEVC, H.266 / VVC, or AVS, the prediction blocks and transform blocks of the same CU may have different sizes and numbers.

[0038] exist Figure 1 In the structure 110, the basic processing unit 112 is further divided into 3×3 basic processing sub-units, and the boundaries of these basic processing sub-units are shown by dotted lines. In different schemes, different basic processing units of the same image can be divided into basic processing sub-units.

[0039] In some embodiments, in order to provide parallel processing and error resilience for video encoding and decoding, an image can be divided into multiple regions for processing, so that for a certain region of the image, the encoding or decoding process can be independent of information from any other region of the image. In other words, each region of the image can be processed independently. By doing so, the codec can process different regions of an image in parallel, thereby improving coding efficiency. In addition, when data in a certain region is corrupted during processing or lost during network transmission, the codec can correctly encode or decode other regions of the same image without relying on the corrupted or lost data, thereby providing error resilience. In some video coding standards, an image can be divided into different types of regions. For example, H.265 / HEVC, H.266 / VVC, and AVS provide two types of regions: "slices" and "tiles." It should also be noted that different images in the video sequence 100 can have different partitioning schemes for dividing an image into multiple regions.

[0040] For example, in Figure 1 In FIG, the structure 110 is divided into three regions 114, 116 and 118, the boundaries of which are shown as solid lines within the structure 110. Region 114 includes four basic processing units. Each of regions 116 and 118 includes six basic processing units. It should be noted that Figure 1 The basic processing units, basic processing sub-units, and regions of the structure 110 are merely examples, and the present disclosure does not limit embodiments thereof.

[0041] Figure 2A Schematic diagram of an example encoding process 200A consistent with an embodiment of the present disclosure is shown. For example, the encoding process 200A may be performed by an encoder. Figure 2A As shown, the encoder may encode the video sequence 202 into a video bitstream 228 according to process 200A. Figure 1 The video sequence 100 in FIG. 2 may include a set of images (referred to as “original images”) arranged in time sequence. Figure 1 Based on the structure 110 in FIG. 1 , the encoder may divide each original image of the video sequence 202 into a plurality of basic processing units, a plurality of basic processing sub-units, or a plurality of regions for processing. In some embodiments, the encoder may perform process 200A at the level of a basic processing unit for each original image of the video sequence 202. For example, the encoder may perform process 200A in an iterative manner, wherein the encoder may encode a basic processing unit in one iteration of process 200A. In some embodiments, the encoder may perform process 200A in parallel for multiple regions (e.g., regions 114-118) of each original image of the video sequence 202.

[0042] exist Figure 2A 2, the encoder may feed the basic processing units of the original images of the video sequence 202 (referred to as "original BPUs") to a prediction stage 204 to generate prediction data 206 and a prediction BPU 208. The encoder may subtract the prediction BPU 208 from the original BPU to generate a residual BPU 210. The encoder may feed the residual BPU 210 to a transform stage 212 and a quantization stage 214 to generate quantized transform coefficients 216. The encoder may feed the prediction data 206 and the quantized transform coefficients 216 to a binary encoding stage 226 to generate a video bitstream 228. Components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 may be referred to as a "forward path." During process 200A, after the quantization stage 214, the encoder may feed the quantized transform coefficients 216 to an inverse quantization stage 218 and an inverse transform stage 220 to generate a reconstructed residual BPU 222. The encoder may add the reconstruction residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224, which is used in the prediction stage 204 for the next iteration of process 200A. Components 218, 220, 222, and 224 of process 200A may be referred to as a "reconstruction path." The reconstruction path may be used to ensure that both the encoder and the decoder use the same reference data for prediction.

[0043] The encoder may iteratively perform process 200A to encode each original BPU of the original image (in the forward path) and generate a prediction reference 224 for encoding the next original BPU of the original image (in the reconstruction path). After encoding all original BPUs of the original image, the encoder may proceed to encode the next image in the video sequence 202.

[0044] Referring to process 200A, the encoder may receive a video sequence generated by a video capture device (e.g., a camera) 202. As used herein, the term "receive" may refer to any action of receiving, inputting, acquiring, retrieving, obtaining, reading, accessing, or inputting data in any manner.

[0045] At the current iteration, at the prediction stage 204, the encoder may receive the original BPU and the prediction reference 224 and perform a prediction operation to generate the prediction data 206 and the predicted BPU 208. The prediction reference 224 may be generated from the reconstruction path of the previous iteration of the process 200A. The purpose of the prediction stage 204 is to reduce information redundancy by extracting the prediction data 206 from the prediction data 206 and the prediction reference 224 that can be used to reconstruct the original BPU into the predicted BPU 208.

[0046] Ideally, the predicted BPU 208 would be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 208 typically differs slightly from the original BPU. To account for such differences, after generating the predicted BPU 208, the encoder may subtract the predicted BPU 208 from the original BPU to generate a residual BPU 210. For example, the encoder may subtract the values ​​of the pixels of the predicted BPU 208 (e.g., grayscale values ​​or RGB values) from the values ​​of the corresponding pixels of the original BPU. Each pixel of the residual BPU 210 may have a residual value generated by this subtraction between the corresponding pixels of the original BPU and the predicted BPU 208. Compared to the original BPU, the predicted data 206 and the residual BPU 210 may have fewer bits, but they can be used to reconstruct the original BPU without noticeable quality degradation. Thus, the original BPU is compressed.

[0047] To further compress the residual BPU 210, in the transform stage 212, the encoder can reduce the spatial redundancy of the residual BPU 210 by decomposing it into a set of two-dimensional "basis patterns", each of which is associated with a "transform coefficient". The basis patterns can have the same size (e.g., the size of the residual BPU 210). Each basis pattern can represent a frequency-varying component of the residual BPU 210 (e.g., the frequency of luminance variation). No basis pattern can be reproduced by any combination (e.g., linear combination) of any other basis patterns. In other words, the decomposition can decompose the variation of the residual BPU 210 into the frequency domain. This decomposition is analogous to the discrete Fourier transform of a function, where the basis patterns are analogous to the basis functions (e.g., trigonometric functions) of the discrete Fourier transform, and the transform coefficients are analogous to the coefficients associated with the basis functions.

[0048] Different transform algorithms can use different base patterns. Various transform algorithms can be used in the transform stage 212, such as discrete cosine transform, discrete sine transform, and the like. The transform in the transform stage 212 is reversible. That is, the encoder can restore the residual BPU 210 by performing the inverse operation of the transform (referred to as an "inverse transform"). For example, to restore a pixel of the residual BPU 210, the inverse transform may involve multiplying the value of the corresponding pixel in the base pattern by the corresponding correlation coefficient and adding the products to produce a weighted sum. For video coding standards, both the encoder and decoder can use the same transform algorithm (and therefore the same base pattern). Therefore, the encoder can only record the transform coefficients, and the decoder can reconstruct the residual BPU 210 based on the transform coefficients without receiving the base pattern from the encoder. Compared to the residual BPU 210, the transform coefficients may have fewer bits, but they can be used to reconstruct the residual BPU 210 without significant quality degradation. Consequently, the residual BPU 210 is further compressed.

[0049] The encoder can further compress the transform coefficients during the quantization stage 214. During the transform process, different basis patterns can represent different frequencies of variation (e.g., the frequency of brightness variation). Because the human eye is generally better at discerning low-frequency variations, the encoder can ignore information about high-frequency variations without causing noticeable quality degradation in decoding. For example, during the quantization stage 214, the encoder can generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (called a "quantization scale factor") and rounding the quotient to the nearest integer. After this operation, some transform coefficients of high-frequency basis patterns may be converted to zero, while the transform coefficients of low-frequency basis patterns may be converted to smaller integers. The encoder can ignore quantized transform coefficients 216 with zero values, thereby further compressing the transform coefficients. The quantization process is also reversible, where the quantized transform coefficients 216 can be reconstructed into the transform coefficients in the inverse operation of quantization (called "inverse quantization").

[0050] Because the encoder ignores the remainder of this division in the rounding operation, the quantization stage 214 may be lossy. In general, the quantization stage 214 may constitute the largest source of information loss in process 200A. The greater the information loss, the fewer bits may be required to quantize the transform coefficients 216. To achieve different degrees of information loss, the encoder may use different values ​​for the quantization parameter or any other parameter of the quantization process.

[0051] In the binary encoding stage 226, the encoder may encode the prediction data 206 and the quantized transform coefficients 216 using a binary encoding technique, such as, for example, entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless or lossy compression algorithm. In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the encoder may also encode other information in the binary encoding stage 226, such as, for example, the prediction mode used in the prediction stage 204, the parameters of the prediction operation, the transform type of the transform stage 212, the parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bit rate control parameters), etc. The encoder may use the output data of the binary encoding stage 226 to generate a video bitstream 228. In some embodiments, the video bitstream 228 may be further packaged for network transmission.

[0052] Referring to the reconstruction path of process 200A, the encoder may perform inverse quantization on the quantized transform coefficients 216 to generate reconstructed transform coefficients in an inverse quantization stage 218. In an inverse transform stage 220, the encoder may generate a reconstructed residual BPU 222 based on the reconstructed transform coefficients. The encoder may add the reconstructed residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224 to be used in the next iteration of process 200A.

[0053] It should be noted that other variations of process 200A may be used to encode video sequence 202. In some embodiments, the stages of process 200A may be performed by the encoder in a different order. In some embodiments, one or more stages of process 200A may be combined into a single stage. In some embodiments, a single stage of process 200A may be divided into multiple stages. For example, transform stage 212 and quantization stage 214 may be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, process 200A may be omitted. Figure 2A one or more stages in a process.

[0054] Figure 2B A schematic diagram of another example encoding process 200B consistent with embodiments of the present disclosure is shown. Process 200B can be modified from process 200A. For example, process 200B can be used by encoders compliant with hybrid video coding standards (e.g., the H.26x series). Compared to process 200A, the forward path of process 200B additionally includes a mode decision stage 230 and divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B additionally includes a loop filter stage 232 and a buffer 234.

[0055] In general, prediction techniques can be divided into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-image prediction or "intra-frame prediction") can use pixels from one or more coded adjacent BPUs in the same image to predict the current BPU. That is, the prediction reference 224 in spatial prediction can include the adjacent BPUs. The spatial prediction can reduce the inherent spatial redundancy of the image. Temporal prediction (e.g., inter-image prediction or "inter-frame prediction") can use regions from one or more coded images to predict the current BPU. That is, the prediction reference 224 in temporal prediction can include coded images. The temporal prediction can reduce the inherent temporal redundancy of the image.

[0056] Referring to process 200B, in the forward path, the encoder performs prediction operations in a spatial prediction stage 2042 and a temporal prediction stage 2044. For example, in the spatial prediction stage 2042, the encoder may perform intra-frame prediction. For a particular original BPU of a particular picture being encoded, the prediction reference 224 may include one or more neighboring BPUs that have been encoded (in the forward path) and reconstructed (in the reconstruction path) in the same picture. The encoder may generate a predicted BPU 208 by extrapolating the neighboring BPUs. The extrapolation techniques may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, etc. In some embodiments, the encoder may perform the extrapolation at the pixel level, for example, extrapolating the corresponding pixel value of each pixel of the predicted BPU 208. The neighboring BPU used for extrapolation can be positioned in various directions relative to the original BPU, such as vertically (e.g., on top of the original BPU), horizontally (e.g., to the left of the original BPU), diagonally (e.g., below left, below right, above left, or above right of the original BPU), or any direction defined in the video coding standard used. For intra-frame prediction, the prediction data 206 may include, for example, the position (e.g., coordinates) of the neighboring BPU used, the size of the neighboring BPU used, parameters of the extrapolation, the direction of the neighboring BPU used relative to the original BPU, etc.

[0057] For another example, in the temporal prediction stage 2044, the encoder may perform the inter-frame prediction. For a certain original BPU of the current image, the prediction reference 224 may include one or more images (referred to as "reference images") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference images may be encoded and reconstructed on a BPU-by-BPU basis. For example, the encoder may add the reconstructed residual BPU 222 to the prediction BPU 208 to generate a reconstructed BPU. After all reconstructed BPUs for the same image are generated, the encoder may generate the reconstructed image as a reference image. The encoder may perform a "motion estimation" operation to search for a matching area within a certain range (referred to as a "search window") of the reference image. The position of the search window in the reference image may be determined based on the position of the original BPU in the current image. For example, the search window may be centered at a location in the reference image that has the same coordinates as the original BPU in the current image and may extend outward by a predetermined distance. When the encoder identifies an area similar to the original BPU in the search window (e.g., by using a pixel recursion algorithm, a block matching algorithm, etc.), the encoder can determine such an area as a matching area. The matching area can have different specifications from the original BPU (e.g., smaller than, equal to, larger than, or having a different shape). Because the reference image and the current image are temporally separated in the time axis (e.g., as shown in FIG. 2 ), the matching area can be a different size than the original BPU. Figure 1 As shown in Figure 2, the matching area can be considered to "move" to the position of the original BPU over time. The encoder can record the direction and distance of this movement as a "motion vector". When using multiple reference images (e.g., Figure 1 When the encoder searches for a matching region for each reference image and determines an associated motion vector for the matching region, the encoder may assign weights to the pixel values ​​of the matching region of each matching reference image.

[0058] The motion estimation may be used to identify various types of motion, such as translation, rotation, scaling, etc. For inter-frame prediction, the prediction data 206 may include, for example, the location (e.g., coordinates) of the matching region, a motion vector associated with the matching region, the number of reference images, weights associated with the reference images, etc.

[0059] To generate the predicted BPU 208, the encoder may perform a "motion compensation" operation. The motion compensation may be used to reconstruct the predicted BPU 208 based on the prediction data 206 (e.g., motion vector) and the prediction reference 224. For example, the encoder may move the matching area of ​​the reference image according to the motion vector, thereby predicting the original BPU of the current image. When using multiple reference images (e.g., Figure 1 When the encoder obtains the image 106 in the reference image, the encoder may move the matching region of the reference image according to the respective motion vectors and average pixel values ​​of the matching regions. In some embodiments, if the encoder has assigned weights to the pixel values ​​of the matching regions of the respective matching reference images, the encoder may perform a weighted summation of the pixel values ​​of the moved matching regions.

[0060] In some embodiments, the inter-frame prediction can be unidirectional or bidirectional. Unidirectional inter-frame prediction can use one or more reference images in the same temporal direction relative to the current image. For example, Figure 1 The picture 104 in is a unidirectional inter-frame predicted picture, wherein the reference picture (e.g., picture 102) precedes the picture 104. Bidirectional inter-frame prediction can use one or more reference pictures in two temporal directions relative to the current picture. For example, Figure 1 The picture 106 in is a bi-directionally inter-predicted picture, wherein the reference pictures (eg, pictures 104 and 108 ) are in both temporal directions relative to picture 104 .

[0061] Still referring to the forward path of process 200B, after the spatial prediction stage 2042 and the temporal prediction stage 2044, in the mode decision stage 230, the encoder can select a prediction mode (e.g., one of intra prediction or inter prediction) for the current iteration of process 200B. For example, the encoder can perform a rate-distortion optimization technique, wherein the encoder can select a prediction mode based on the bit rate of a candidate prediction mode and the distortion of a reference image reconstructed under the candidate prediction mode to minimize the value of a cost function. Based on the selected prediction mode, the encoder can generate a corresponding prediction BPU 208 and prediction data 206.

[0062] In the reconstruction path of process 200B, if intra prediction mode has been selected in the forward path, after generating the prediction reference 224 (e.g., the current BPU in the current picture that has been encoded and reconstructed), the encoder can feed the prediction reference 224 directly to the spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU of the current picture). The encoder can feed the prediction reference 224 to the loop filtering stage 232, where the encoder can apply loop filtering to the prediction reference 224 to reduce or eliminate distortion (e.g., blocking artifacts) introduced during the encoding of the prediction reference 224. The encoder can apply various loop filtering techniques in the loop filtering stage 232, such as deblocking, sample adaptive offset, adaptive loop filtering, etc. The loop-filtered reference picture can be stored in a buffer 234 (or "decoded picture buffer") for subsequent use (e.g., as an inter-frame prediction reference picture for a future picture in the video sequence 202). The encoder can store one or more reference pictures in the buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the encoder may encode loop filtering parameters (eg, loop filtering strength) as well as quantized transform coefficients 216 , prediction data 206 , and other information in a binary encoding stage 226 .

[0063] Figure 3A A schematic diagram of an example decoding process 300A consistent with an embodiment of the present disclosure is shown. Process 300A may correspond to Figure 2A In some embodiments, process 300A may be similar to the reconstruction path of process 200A. The decoder may decode the video bitstream 228 into a video stream 304 according to process 300A. Video stream 304 may be very similar to video sequence 202. However, due to information loss during compression and decompression (e.g., Figures 2A-2B quantization stage 214 in), typically, the video stream 304 is different from the video sequence 202. Figures 2A-2B 200A and 200B, the decoder may perform process 300A at the basic processing unit (BPU) level for each picture encoded in the video bitstream 228. For example, the decoder may perform process 300A in an iterative manner, wherein the decoder may decode a basic processing unit in one iteration of process 300A. In some embodiments, the decoder may perform process 300A in parallel for multiple regions (e.g., regions 114-118) of each picture encoded in the video bitstream 228.

[0064] exist Figure 3AIn the process 300A, the decoder may feed a portion of the video bitstream 228 associated with the basic processing unit of the encoded picture (referred to as the "encoded BPU") to the binary decoding stage 302. In the binary decoding stage 302, the decoder may decode the portion into prediction data 206 and quantized transform coefficients 216. The decoder may feed the quantized transform coefficients 216 to the inverse quantization stage 218 and the inverse transform stage 220 to generate a reconstructed residual BPU 222. The decoder may feed the prediction data 206 to the prediction stage 204 to generate the prediction BPU 208. The decoder may add the reconstructed residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224. In some embodiments, the prediction reference 224 may be stored in a buffer (e.g., a decoded picture buffer in computer memory). The decoder may feed the prediction reference 224 to the prediction stage 204 for use in performing a prediction operation in the next iteration of the process 300A.

[0065] The decoder may iteratively perform process 300A to decode each coded BPU of the coded picture and generate a prediction reference 224 for encoding the next coded BPU of the coded picture. After decoding all coded BPUs of the coded picture, the decoder may output the picture to a video stream 304 for display and continue decoding the next coded picture in the video bitstream 228.

[0066] In the binary decoding stage 302, the decoder may perform the inverse of the binary coding technique used by the encoder (e.g., entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the decoder may also decode other information in the binary decoding stage 302, such as, for example, the prediction mode, parameters of the prediction operation, the transform type, parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), etc. In some embodiments, if the video bitstream 228 is transmitted over a network in the form of packets, the decoder may depacketize the video bitstream 228 before feeding it to the binary decoding stage 302.

[0067] Figure 3BA schematic diagram of another example decoding process 300B consistent with embodiments of the present disclosure is shown. Process 300B may be modified from process 300A. For example, process 300B may be used by a decoder compliant with a hybrid video coding standard (e.g., the H.26x series). Compared to process 300A, process 300B additionally divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and additionally includes a loop filter stage 232 and a buffer 234.

[0068] In process 300B, for an encoded basic processing unit (referred to as a "current BPU") of an encoded image being decoded (referred to as a "current image"), the prediction data 206 decoded by the decoder from the binary decoding stage 302 may include various types of data, depending on the prediction mode used by the encoder to encode the current BPU. For example, if the encoder used intra-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating intra-frame prediction, parameters of the intra-frame prediction operation, etc. The parameters of the intra-frame prediction operation may include, for example, the location (e.g., coordinates) of one or more neighboring BPUs used as reference, the size of the neighboring BPUs, extrapolation parameters, the orientation of the neighboring BPUs relative to the original BPU, etc. For another example, if the encoder used inter-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating inter-frame prediction, parameters of the inter-frame prediction operation, etc. The parameters of the inter-frame prediction operation may include, for example, the number of reference images associated with the current BPU, the weights associated with the reference images respectively, the positions (e.g., coordinates) of one or more matching regions in the respective reference images, one or more motion vectors associated with the matching regions respectively, etc.

[0069] Based on the prediction mode indicator, the decoder may decide whether to perform spatial prediction (eg, intra prediction) in a spatial prediction stage 2042 or temporal prediction (eg, inter prediction) in a temporal prediction stage 2044 . Figure 2B The details of performing such spatial prediction or temporal prediction are described in detail in

[15] and will not be repeated here. After performing such spatial prediction or temporal prediction, the decoder may generate a prediction BPU 208. The decoder may add the prediction BPU 208 and the reconstructed residual BPU 222 to generate a prediction reference 224, as shown in FIG. Figure 3A described.

[0070] In process 300B, the decoder may feed the prediction reference 224 to the spatial prediction stage 2042 or the temporal prediction stage 2044 for performing a prediction operation in the next iteration of process 300B. For example, if the current BPU is decoded using the intra prediction in the spatial prediction stage 2042, then after generating the prediction reference 224 (e.g., the decoded current BPU), the decoder may feed the prediction reference 224 directly to the spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU of the current picture). If the current BPU is decoded using the inter prediction in the temporal prediction stage 2044, then after generating the prediction reference 224 (e.g., the reference picture in which all BPUs have been decoded), the decoder may feed the prediction reference 224 to the loop filtering stage 232 to reduce or eliminate distortion (e.g., blocking artifacts). The decoder may perform the following operations: Figure 2B In-loop filtering is applied to the prediction reference 224 in the manner described in

[15] . The loop-filtered reference picture can be stored in a buffer 234 (e.g., a decoded picture buffer in a computer memory) for subsequent use (e.g., as an inter-frame prediction reference picture for a future encoded picture in the video bitstream 228). The decoder can store one or more reference pictures in the buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the prediction data can further include loop filtering parameters (e.g., loop filtering strength). In some embodiments, when the prediction mode indicator of the prediction data 206 indicates that inter-frame prediction was used to encode the current BPU, the prediction data includes loop filtering parameters.

[0071] Figure 4 is a block diagram of an example apparatus 400 for encoding or decoding video consistent with an embodiment of the present disclosure. Figure 4As shown, the device 400 may include a processor 402. When the processor 402 executes the instructions described herein, the device 400 may become a special-purpose machine for video encoding or decoding. The processor 402 may be any type of circuit system capable of manipulating or processing information. For example, the processor 402 may include any number of central processing units (or "CPUs"), graphics processing units (or "GPUs"), neural processing units ("NPUs"), microcontroller units ("MCUs"), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLAs), programmable array logic (PALs), general array logic (GALs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), systems on chip (SoCs), application-specific integrated circuits (ASICs), and the like. In some embodiments, the processor 402 may also be a group of processors grouped into a single logical component. For example, as Figure 4 As shown, processor 402 may include multiple processors, including processor 402a, processor 402b, and processor 402n.

[0072] The apparatus 400 may also include a memory 404 configured to store data (eg, instruction sets, computer code, intermediate data, etc.). Figure 4 As shown, the stored data may include program instructions (e.g., program instructions for implementing the stages in process 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequence 202, video bitstream 228, or video stream 304). Processor 402 may access the program instructions and data for processing (e.g., via bus 410) and execute the program instructions to perform operations or manipulations on the data for processing. Memory 404 may include a high-speed random access memory device or a non-volatile memory device. In some embodiments, memory 404 may include any combination of any number of random access memories (RAM), read-only memories (ROM), optical disks, magnetic disks, hard drives, solid-state drives, flash drives, secure digital (SD) cards, memory sticks, compact flash (CF) cards, etc. Memory 404 may also be a group of memories grouped into a single logical component (e.g., a memory card). Figure 4 not shown).

[0073] The bus 410 may be a communication device for transmitting data between components within the apparatus 400 , such as an internal bus (eg, a CPU-memory bus), an external bus (eg, a Universal Serial Bus port, a Peripheral Component Interconnect Express port), and the like.

[0074] For ease of explanation and to avoid ambiguity, the processor 402 and other data processing circuitry are collectively referred to as "data processing circuitry" in this disclosure. The data processing circuitry may be implemented entirely in hardware, or as a combination of software, hardware, or firmware. Furthermore, the data processing circuitry may be a single standalone module, or may be fully or partially integrated into any other component of the device 400.

[0075] The device 400 may also include a network interface 406 to provide wired or wireless communication with a network (e.g., the Internet, an intranet, a local area network, a mobile communication network, etc.). In some embodiments, the network interface 406 may include any number of network interface controllers (NICs), radio frequency (RF) modules, transponders, transceivers, modems, routers, gateways, wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication ("NFC") adapters, cellular network chips, etc.

[0076] In some embodiments, the apparatus 400 may optionally further include a peripheral interface 408 to provide a connection to one or more peripheral devices. Figure 4 As shown, peripheral devices may include, but are not limited to, a cursor control device (e.g., a mouse, touchpad, or touch screen), a keyboard, a display (e.g., a cathode ray tube display, a liquid crystal display, or a light emitting diode display), a video input device (e.g., a camera or an input interface coupled to a video archive), etc.

[0077] It should be noted that a video codec (e.g., a codec that performs processes 200A, 200B, 300A, or 300B) can be implemented as any combination of software or hardware modules in device 400. For example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more software modules of device 400, such as program instructions that can be loaded into memory 404. For another example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more hardware modules of device 400, such as dedicated data processing circuits (e.g., FPGAs, ASICs, NPUs, etc.).

[0078] The VVC standard uses the Bidirectional Optical Flow (BDOF) tool. BDOF, previously known as Bidirectional Optical Flow, was proposed to improve the accuracy of inter-frame prediction based on optical flow and was first adopted in the JEM software. Compared to the JEM version, the BDOF in VVC is a simplified version that requires less calculation, especially in terms of the number of multiplication operations and the size of the multipliers.

[0079] In VVC, BDOF is used to refine the bidirectional prediction signal of a CU at the 4×4 sub-block level. BDOF is applied to a CU if it meets all of the following conditions: - The CU is encoded using "true" bi-prediction mode, i.e. one of the two reference pictures precedes the current picture in display order and the other follows the current picture in display order - The distance from the two reference images to the current image (i.e., POC difference) is the same - Both reference images are short-term reference images - Do not use affine mode or SbTMVP merge mode to encode the CU -CU has more than 64 luma samples -CU height and CU width are both greater than or equal to 8 luma samples - Bi-prediction with CU-level weight (BCW) weight index indicates equal weight - Weighted Prediction (WP) is not enabled for the current CU - Combined inter and intra prediction (CIIP) mode is not used for the current CU

[0080] BDOF is applied only to the luma component. As the name suggests, the BDOF mode is based on the optical flow concept, which assumes that the motion of the object is smooth. For each 4×4 sub-block, the motion improvement (v x ,v y Then, the motion refinement is used to adjust the bidirectional prediction sample values ​​in the 4×4 sub-block. The following steps are applied in the BDOF process.

[0081] First, the horizontal gradient of the two prediction signals is calculated by directly computing the difference between two adjacent samples and vertical gradient (k=0,1), that is, Among them, I (k) (i, j) is the sample value at coordinate (i, j) of the prediction signal in list k (k=0, 1), and shift1 is calculated based on the luma bit depth bitDepth as shift1=max(6, bitDepth-6).

[0082] Then, the autocorrelations and cross-correlations of the gradients S1, S2, S3, S5, and S6 are calculated as in, where Ω is a 6×6 window around the 4×4 sub-block, and n a and n b The values ​​of are set equal to min(1, bitDepth-11) and min(4, bitDepth-8), respectively, where min(x, y) is a function that obtains the smaller of x and y.

[0083] Then, the motion improvement (v) is derived using the cross-correlation term and the autocorrelation term according to the following formula: x ,v y ): in, th′ BIO =2 max(5,BD-7) . is a floor function, and

[0084] Based on the motion refinement and gradient, the following adjustment is calculated for each sample in the 4×4 sub-block:

[0085] Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as follows: pred BDOF (x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+O offset )>>shift (6)

[0086] These values ​​are selected so that the multiplier in the BDOF process does not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process remains within 32 bits.

[0087] In order to derive the gradient value, it is necessary to generate some prediction samples I in the list k (k=0,1) outside the current CU boundary. (k) (i,j). Figure 5As depicted, the BDOF in VVC uses an extended row / column around the CU boundary. To control the computational complexity of generating prediction samples outside the boundary, prediction samples in the extended region (white locations) are generated by directly accessing reference samples at adjacent integer locations (using a floor() operation on the coordinates) without interpolation, and a conventional 8-tap motion-compensated interpolation filter is used to generate prediction samples within the CU (gray locations). These extended sample values ​​are used only for gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values ​​outside the CU boundary are needed, they are padded (i.e., repeated) from their nearest neighbors.

[0088] When the width or height of a CU is greater than 16 luma samples, it will be split into sub-blocks with a width or height equal to 16 luma samples, and the sub-block boundaries are regarded as the CU boundaries in the BDOF process. The maximum unit size for the BDOF process is limited to 16×16. The BDOF process can be skipped for each sub-block. When the SAD (Sum of Absolute Differences) between the initial L0 and L1 prediction samples is less than a threshold, the BDOF is not applied to the sub-block. The threshold is set to equal to (8*W*(H>>1), where W indicates the sub-block width and H indicates the sub-block height. In order to avoid the additional complexity of SAD calculation, the SAD between the initial L0 and L1 prediction samples calculated in the DMVR process is reused here.

[0089] If BCW is enabled for the current block, i.e., the BCW weight index indicates unequal weights, then bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, i.e., luma_weight_lx_flag is 1 for either of the two reference images, then BDOF is also disabled. BDOF is also disabled when the CU is encoded in symmetric MVD mode or CIIP mode.

[0090] In ECM, in order to further improve the coding performance, a sample-based BDOF is adopted. In the sample-based BDOF, motion improvement (v x ,v y ), but rather execute on a sample basis.

[0091] In addition to sample-level motion refinement calculations, for samples outside the CU boundary, true sample values ​​are used instead of padded values. Therefore, the decoder needs to retrieve and interpolate sample regions larger than the current CU for gradient calculations. Similarly, since true sample values ​​are used for samples outside the boundary, there is no need to split the CU into 16×16 sub-blocks for boundary padding.

[0092] For threshold checking, the coding block is divided into 8×8 sub-blocks. For each sub-block, the SAD between two reference sub-blocks is checked against a threshold to determine whether to apply BDOF. If it is decided to apply BDOF to a sub-block, a sliding 5×5 window is used for each sample in the sub-block, and the existing BDOF process is applied to each sliding window to derive v x and v y Applying the derived motion refinement (v x ,v y ) to adjust the bidirectional prediction sample value of the center sample of the window.

[0093] Since the 8-tap interpolation filter used in the VVC is replaced by a 12-tap filter, BDOF also uses a 12-tap interpolation filter to generate the prediction samples.

[0094] Next, high-precision BDOF is described. To improve the accuracy of the motion improvement calculation, the following formula is used instead of formula (4).

[0095] First, the horizontal gradient of the two prediction signals is calculated by directly calculating the difference between two adjacent samples and vertical gradient (k=0,1), that is, Among them, I (k) (i, j) is the sample value at coordinate (i, j) of the prediction signal in list k (k=0, 1), and shift1 is calculated based on the luma bit depth.

[0096] Then, the autocorrelations and cross-correlations of the gradients S1, S2, S3, S4, S5, and S6 are calculated as in, Among them, w ij is the weight associated with position (i, j). The closer the position (i, j) is to the current sample, the ij The larger the value of , Ω is a 5×5 window around the current sample, and n a and n b The values ​​of are set equal to min(1, bitDepth-11) and min(4, bitDepth-8), respectively, where min(x, y) is a function that obtains the smaller of x and y.

[0097] In some embodiments, by extracting motion improvements (v x ,v y ) before applying the adjustment operation by adding the offset to S1 and S5. The adjustment operation can be expressed as: Wherein, d is the offset, and its value may depend on the sub-block size.

[0098] Then, the motion improvement (v) is derived using the cross-correlation term and the autocorrelation term according to the following formula: x ,v y ):

[0099] Based on the motion improvement and the gradient, the following adjustment is calculated for each sample:

[0100] Finally, the BDOF samples of the CU are calculated by adjusting the bidirectional prediction samples as follows: pred BDOF (x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset )>>shift (13)

[0101] Next, the decoding-side motion vector improvement (DMVR) is described. VVC uses the decoding-side motion vector improvement based on bilateral matching (BM) in the bidirectional prediction operation to increase the accuracy of the MV of the merge mode. In DMVR, the improved MV is searched around the initial MV in the reference image list L0 and the reference image list L1. The BM method calculates the distortion between the two candidate blocks in the reference image list L0 and the list L1. Figure 6 As shown, the SAD between red blocks is calculated based on each MV candidate around the initial MV. The MV candidate with the lowest SAD becomes the improved MV and is used to generate the bidirectional prediction signal.

[0102] In VVC, the application of DMVR is restricted and applies only to the CUs coded with the following modes and features: -CU level merge mode with bi-predictive MV - With respect to the current image, one reference image is located in the past and the other reference image is located in the future - The distances from the two reference images to the current image (i.e., POC differences) are the same - Both reference images are short-term reference images -CU has more than 64 luma samples -CU height and CU width are both greater than or equal to 8 luma samples -BCW weight index indicates equal weight - WP is not enabled for the current block - CIIP mode is not used for the current block

[0103] The modified MV derived by the DMVR process is used to generate the inter-frame prediction samples and is also used for temporal motion vector prediction of future image coding, while the original MV is used in the deblocking process and is also used for spatial motion vector prediction of future CU coding.

[0104] Additional features of DMVR are mentioned in the following subclauses.

[0105] Next, the search scheme used in DMVR is described. In DMVR, the search point is centered around the initial MV, and the MV offset follows the MV difference mirror rule. In other words, any point represented by a candidate MV pair (MV0, MV1) examined by DMVR follows the following two formulas: MV0′=MV0+MV_offset (14) MV1′=MV1-MV_offset (15) Wherein, MV_offset represents the modified offset between the initial MV and the modified MV in one of the reference images. The modified search range is two integer luminance samples from the initial MV. The search includes an integer sample offset search phase and a fractional sample modification phase.

[0106] A 25-point full search is applied for integer sample offset search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. To reduce the uncertainty penalty of DMVR improvement, it is proposed to give priority to the original MV in the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by 1 / 4 of the SAD value.

[0107] The integer sample search is followed by a fractional sample refinement. To reduce computational complexity, the fractional sample refinement is derived using a parametric error surface formula rather than an additional search using SAD comparisons. The fractional sample refinement is conditionally invoked based on the output of the integer sample search phase. If the integer sample search phase finds a center point with a minimum SAD value in either the first or second iteration of the search, the search phase is terminated and the fractional sample refinement is not further applied.

[0108] In the sub-pixel offset estimation based on the parameter error surface, the center position cost and the costs at four positions adjacent to the center are used to fit a 2-D parabolic error surface formula of the following form: E(x,y)=A(xx min ) 2 +B(yy min ) 2 +C (16) Among them, (x min ,y min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above formula using the cost values ​​of the five search points, (x min ,y min ) is calculated as: x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (17) y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (18)

[0109] x min and y min The value of is automatically clamped between -8 and 8, since all cost values ​​are positive and the minimum value is E(0,0). This corresponds to a half-chime offset with 1 / 16 pixel MV accuracy in VVC. The calculated fraction (x min ,y min ) is added to the integer distance improved MV to obtain a sub-pixel accurate improved delta MV.

[0110] Next, bilinear interpolation and sample padding are described. In VVC, the resolution of the MV is 1 / 16 luma samples. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search point surrounds the initial fractional pixel MV with integer sample offsets, so samples at these fractional positions need to be interpolated for the DMVR search process. To reduce computational complexity, a bilinear interpolation filter is used to generate the fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter, the DMVR process does not access more reference samples within the 2-sample search range compared to the normal motion compensation process. After obtaining the improved MV using the DMVR search process, a conventional 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples in the normal MC process, samples that are not required by the interpolation process based on the original MV but are required by the interpolation process based on the improved MV are padded from available samples.

[0111] Next, the maximum DMVR processing unit is described. When the width or height of a CU is greater than 16 luma samples, it will be further split into sub-blocks with a width or height equal to 16 luma samples. The maximum unit size of the DMVR search process is limited to 16×16.

[0112] Next, we describe multi-pass decoder-side motion vector refinement (MP-DMVR). To further improve coding efficiency in ECM, multiple passes of decoder-side motion vector refinement are applied. In the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16×16 sub-block within the coding block. In the third pass, bidirectional optical flow (BDOF) is applied to refine the MVs in each 8×8 sub-block. The refined MVs are stored for use in both spatial and temporal motion vector prediction.

[0113] In the first pass, block-based bilateral matching MV refinement is performed. In the first pass, a refined MV is derived by applying BM to the coded block. Similar to Decoder-side Motion Vector Refinement (DMVR), in a bi-prediction operation, a refined MV is searched around the two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. Based on the minimum bilateral matching cost between the two reference blocks in L0 and L1, refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MVs.

[0114] BM performs a local search to derive integer sample precision intDeltaMV. The local search applies a 3×3 square search pattern to cycle through the search range [–sHor, sHor] in the horizontal direction and [–sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension and the maximum values ​​of sHor and sVer are 8 or other values. For example, as in Figure 7 In the example, point 0 is the position pointed to by the initial MV. Therefore, points 1 to 8 around the initial position are first searched, and the cost of each position is calculated. If point 7 has the minimum cost, point 7 is set as the search center, and points 9, 10, and 11 are searched. If the cost of point 10 is less than that of point 7, the search center is shifted to point 10, and points 12, 13, and 14 are searched. If point 12 has the minimum cost among points 6 to 14, point 12 is set as the new search center. If points 10, 11, 13, 15 to 19 around point 10 are all greater than point 10, point 12 is the best position, and the search process stops.

[0115] The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost, where sadCost is the SAD between the l0 prediction value and the l1 prediction value at each search point, and mvDistanceCost is based on intDeltaMV (i.e., the distance between the search point and the initial position). When the block size cbW*cbH is greater than 64, the MRSAD cost function is applied to remove the DC effect of the distortion between reference blocks. When the bilCost at the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV local search terminates. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the search for the minimum cost continues until it reaches the end of the search range.

[0116] The existing fractional sample refinement is further applied to derive the final deltaMV. The refined MV after the first round is then derived as:

[0117] In the second pass, subblock-based bilateral matching MV refinement is performed. In the second pass, a refined MV is derived by applying BM to 16×16 matrix subblocks. For each subblock, a refined MV is searched for around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass in the reference image lists L0 and L1. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference subblocks in L0 and L1.

[0118] For each subblock, BM performs a full search to derive integer sample precision intDeltaMV(sbIdx2). The full search has a search range of [–sHor, sHor] in the horizontal direction and a search range of [–sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimensions and the maximum values ​​of sHor and sVer are 8 or other values.

[0119] The bilateral matching cost is calculated by applying the cost factor to the SATD cost between two reference sub-blocks as: bilCost = satdCost * costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into at most 5 diamond search areas, such as Figure 8 As shown. Each search area is assigned a costFactor, which is determined by the distance intDeltaMV(sbIdx2) between each search point and the starting MV, and each diamond area is processed in order starting from the center of the search area. In each area, the search points are processed in a raster scan order starting from the upper left corner to the lower right corner of the area. When the minimum bilCost in the current search area is less than a threshold equal to sbW*sbH, the inter-pixel full search is terminated, otherwise, the next search area continues the inter-pixel full search until all search points are checked. In addition, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than a threshold equal to the area of ​​the block, the search process is terminated.

[0120] The existing VVC DMVR score sample improvement is further applied to derive the final deltaMV (sbIdx2). Then, the improved MV in the second round is derived as: Where sbIdx2 is the sub-block index of the second round of multi-round DMVR

[0121] In the third round, a sub-block based bidirectional optical flow MV refinement is performed. In the third round, a modified MV is derived by applying BDOF to the 8×8 matrix sub-blocks. For each 8×8 sub-block, the BDOF refinement process is applied starting with the modified MV of the parent sub-block in the second round, and the motion refinement (v) is derived without clipping. x ,v y ) can be improved. The above method can be used. For example, calling formulas (1) to (4) or formulas (7) to (10) to derive (v x ,v y ), denoted as bioMV. After motion-corrected derivation, bioMV is rounded to 1 / 16 sample precision and clipped between -32 and 32.

[0122] The improved MVs (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) in the third round are derived as follows: Wherein, sbIdx2 is the sub-block index of the second round of multiple rounds of DMVR, and sbIdx3 is the sub-block index of the third round of multiple rounds of DMVR.

[0123] Next, the adaptive decoding-side motion vector improvement is described. In ECM, the adaptive decoding-side motion vector improvement method is an extension of multi-round DMVR, which consists of two new merge modes to improve the MV in only one direction (L0 or L1) of the bidirectional prediction for merge candidates that meet the DMVR conditions. The multi-round DMVR process is applied to the selected merge candidates to improve the motion vector, however, MVD0 or MVD1 is set to zero in the first round (i.e., PU level) DMVR. Therefore, a new merge candidate list is constructed for adaptive decoding-side motion vector improvement. And, the new merge mode for the new merge candidate list is called BM merging in ECM.

[0124] The merge candidates for BM merge mode are derived from spatially adjacent coding blocks, TMVP, non-adjacent blocks, history-based motion vector predictor (HMVP), and paired candidates, similar to the conventional merge mode. The difference is that only those candidates that meet the DMVR conditions are added to the candidate list. The two new merge modes use the same merge candidate list. If the BM candidate list contains inherited BCW weights and the DMVR process remains unchanged, but if the weights are not equal and the bidirectional prediction is weighted using the BCW weights, MRSAD or MRSATD is used to calculate the distortion. The merge index is encoded as in the conventional merge mode.

[0125] In current multi-pass DMVR, the first and second passes are based on bilateral matching. CU-level bilateral matching is performed in the first pass, followed by sub-block-level bilateral matching in the second pass. CU-level bilateral matching refines the motion vectors at a coarse level, while sub-block-level bilateral matching refines the motion vectors at a finer level. However, for the third pass of multi-pass DMVR based on optical flow, the refinement matrix is ​​fixed and only one pass is performed. Smaller sub-block-level refinements result in finer refinements, while larger sub-block-level refinements provide greater error tolerance and, therefore, more stable results in the optical flow calculation. Therefore, performing only one pass of optical flow-based refinement on the fixed matrix does not yield accurate and stable results. Furthermore, the optimal sub-block size for motion vector refinement varies dynamically depending on the video content. For example, smooth content may prefer refinement on large sub-blocks, while complex content may require refinement on a small block level. Therefore, motion vector refinement on a fixed matrix is ​​not suitable for all video content.

[0126] The present disclosure provides a method for solving the above problems. In particular, it is proposed to perform multiple rounds of processing on motion vector improvement based on optical flow. First, the CU is divided into large sub-blocks, and the improvement is performed at the large sub-block level. Afterwards, the large sub-block is further divided into smaller sub-blocks, and based on the results of the first round of improvement, a second round of improvement is performed at the smaller sub-block level. In this method, the first round of improvement gives a basic improvement to the large sub-block, and then for each smaller sub-block within the large sub-block, a more accurate motion vector improvement can be obtained by the second round of improvement. For smooth video content, the first round of improvement can significantly improve the accuracy of the motion vector, while for complex video content, the second round of improvement will bring greater benefits. Therefore, both smooth video content and complex video content can benefit from the proposed method.

[0127] However, refinement at the smaller sub-block level introduces more complexity. To reduce complexity, no further sub-block splitting is performed after the first round of refinement at the large sub-block level. That is, the second round of refinement is performed on sub-blocks of the same size as the first round of refinement. In this case, after the CU is divided into the sub-blocks, two rounds of refinement are performed on each sub-block. The first round of refinement derives a first motion refinement, and based on the refined motion, motion compensation is performed on each sub-block, and the optical flow-based refinement is performed again to find a second motion refinement. The final refined motion is obtained by adding the first and second motion refinements.

[0128] Figure 9is a flow chart of an exemplary method 900 for performing motion vector improvement and motion compensation according to some embodiments of the present disclosure. The method 900 may be performed by an encoder (e.g., by Figure 2A Process 200A or Figure 2B 200B) or by a decoder (e.g., by Figure 3A Process 300A or Figure 3B 300B) is performed by the device ( Figure 4 For example, one or more processors (e.g., Figure 4 The method 900 may be performed by a processor 402 of the computer. In some embodiments, the method 900 may be implemented by a computer program product implemented in a computer-readable medium, the computer program product comprising a computer ( Figure 4 Computer executable instructions executed by the device 400) such as program codes. Figure 9 As shown, method 900 includes the following steps 910-950.

[0129] At step 910, the processor performs block-level bilateral matching on the coded block, and the block-level bilateral matching may be the first pass in the multi-pass decoder-side motion vector refinement (MP-DMVR) process.

[0130] At step 920, the processor divides the coding block into N×N sub-blocks and performs sub-block-level bilateral matching. This step can be the second round of the MP-DMVR described above. For example, the coding block can have a size of 64×64, and the processor can divide the coding block into a matrix of 16×16 sub-blocks. The processor can then apply the bilateral matching to multiple 16×16 sub-blocks.

[0131] At step 930, the processor divides the coding block into L×L sub-blocks and performs optical flow-based motion vector refinement on the L×L sub-blocks. This step can be the third round of the MP-DMVR described above, namely the sub-block-level bidirectional optical flow (BDOF) motion vector refinement process. For example, the coding block can have a size of 64×64, and the processor can divide the coding block into a matrix of 16×16 or 8×8 sub-blocks. The processor can then apply the BDOF motion vector refinement process to multiple 16×16 or 8×8 sub-blocks.

[0132] At step 940, the processor divides the coding block into M×M sub-blocks and performs optical flow-based motion vector refinement on the M×M sub-blocks. Consistent with the disclosed embodiments, the processor may repeat the third round of MP-DMVR multiple times on sub-blocks of different sizes. Specifically, at step 940, each of the M×M sub-blocks is smaller than each of the L×L sub-blocks, i.e., L>M. For example, the L×L sub-blocks may be 8×8 sub-blocks, while the M×M sub-blocks may be 4×4 or 2×2 sub-blocks.

[0133] At step 950, the processor performs motion compensation using sample-based bidirectional optical flow.

[0134] Although method 900 performs the BDOF motion vector refinement process twice (i.e., performs the third round of MP-DMVR twice), it is contemplated that the disclosed embodiments may perform this process any number of times, with each round performed on sub-blocks of different sizes. For example, method 900 may be modified to include an additional step 942 ( Figure 9 (not shown). At step 942, the processor may divide the coding block into a plurality of 2×2 sub-blocks, and perform a round of the BDOF motion vector improvement process on the plurality of 2×2 sub-blocks.

[0135] Figure 10 is a flow chart of an exemplary method 1000 for performing motion vector improvement and motion compensation according to some embodiments of the present disclosure. The method 1000 may be performed by an encoder (e.g., by Figure 2A Process 200A or Figure 2B 200B) or by a decoder (e.g., by Figure 3A Process 300A or Figure 3B 300B) is performed by the device ( Figure 4 For example, one or more processors (e.g., Figure 4 The method 1000 may be performed by a processor 402 of the computer. In some embodiments, the method 1000 may be implemented by a computer program product implemented in a computer-readable medium, the computer program product comprising a computer program executed by a computer ( Figure 4 Computer-executable instructions, such as program codes, executed by the device 400).

[0136] like Figure 10As shown, method 1000 includes steps 1010-1050. Steps 1010, 1020, and 1050 are the same as steps 910, 920, and 950, respectively. Method 1000 differs from method 900 in that a first round of BDOF motion vector improvement is first performed on the M×M sub-block (step 1030), and then a second round of BDOF motion vector improvement is performed on the L×L sub-block (step 1040), where L>M.

[0137] Additionally, in some embodiments, adaptive sub-block size can be used for optical flow-based motion vector refinement. For example, Figure 11 is a flow chart of an exemplary method 1100 for performing motion vector improvement and motion compensation according to some embodiments of the present disclosure. The method 1100 may be performed by an encoder (e.g., by Figure 2A Process 200A or Figure 2B 200B) or by a decoder (e.g., by Figure 3A Process 300A or Figure 3B 300B) is performed by the device ( Figure 4 For example, one or more processors (e.g., Figure 4 The method 1100 may be performed by a computer program product implemented in a computer-readable medium. The computer program product includes a computer program product ( Figure 4 Computer executable instructions executed by the device 400) such as program codes. Figure 11 As shown, method 1100 includes the following steps 1110-1160.

[0138] Specifically, step 1110 is the same as steps 910 and 1010, and step 1120 is the same as steps 920 and 1020, and the details are not repeated here.

[0139] At step 1130, the processor checks the sub-block size (generated by step 1120) by comparing it with the condition. If the condition is met, the processor performs optical flow-based motion vector refinement at the large sub-block level (e.g., L×L sub-block) (step 1140); and if the condition is not met, the processor performs optical flow-based motion vector refinement at the small sub-block level (e.g., M×M sub-block) (step 1150). In the disclosed embodiment, the condition can be based on the area of ​​the coding block, the quantization parameter, or the video resolution. For example, the condition can be one of the following: encoding the coding block using a bi-prediction mode; the coding block having a size exceeding a predetermined threshold; the coding block having a size less than a predetermined threshold; disabling weighted prediction for the coding block; disabling joint inter-frame and intra-frame prediction (CIIP) for the coding block; enabling local illumination compensation for the coding block; applying sub-block motion compensation to the coding block; not using symmetric motion vector difference; or not encoding the coding block in merge mode with motion vector difference.

[0140] In addition, at step 1160 , the processor performs motion compensation using sample-based bidirectional optical flow. Step 1160 is the same as steps 950 and 1050 .

[0141] The algorithm for implementing the disclosed optical flow-based motion improvement method is described in detail below. Specifically, in some embodiments, after the second round of multiple rounds of DMVR, the improved motion vector MV0 is obtained for each 16×16 sub-block. pass2 and MV1 pass2 , among which MV0 pass2 is the motion vector of reference picture list 0, and MV1 pass2 is the motion vector of reference picture list 1. Then, based on the improved motion vector MV0 pass2 and MV1 pass2 , perform inter-frame prediction, and obtain Pred0_sb16ext and Pred1_sb16ext, where Pred0_sb16ext is obtained by using MW0 pass2 The block predicted from reference picture list 0, and Pred1_sb16ext is obtained by using MW1 pass2 The block predicted from reference picture list 1. In the inter-frame prediction, an interpolation filter can be applied to generate the predicted pixel value at the sub-pixel position. An 8-tap, 12-tap or other interpolation filter can be used. Since the gradient is required in the optical flow formula, the prediction block is expanded. In this example, Figure 12AAs shown in FIG. 1 , 3 pixel rows or 3 pixel columns are extended on the four boundaries of the 16×16 sub-block. In some other embodiments, in order to reduce bandwidth and interpolation calculation, the extended pixels (i.e., the pixels outside the sub-block boundaries) can be filled from the boundary pixels within the 16×16 sub-block instead of performing actual interpolation processing on them (which is in FIG. 1 ). Figure 5 In some embodiments, to save computation, two adjacent 16×16 modified motion vectors are checked. If two adjacent 16×16 sub-blocks have the same modified motion vector (i.e., MV0 pass2 (sbIdx2) equals MV0 pass2 (sbIdx1), and MV1 pass2 (sbIdx2)isequal toMV1 pass2 (sbIdx2)), two 16×16 adjacent blocks can be merged into a 16×32 or 32×16 sub-block. Then, the motion vector of the merged sub-block is continuously compared with the improved motion vector of the new adjacent 16×16 sub-block. If the motion vectors are still the same, the two sub-blocks are further merged. The merging process is repeated to find the largest possible sub-block. Then, the inter-frame prediction is performed on the merged sub-block. Since the first round of optical flow-based motion improvement is performed at the 8×8 sub-block level, in some other embodiments, the inter-frame prediction can also be performed separately for each 8×8 sub-block within the 16×16 sub-block. This will not affect the result.

[0142] After obtaining the two prediction blocks, the optical flow-based motion vector improvement method can be applied to each 8×8 sub-block in the 16×16 sub-block. For example, the first round of motion improvement MV can be derived using formulas (1) to (4) or formulas (7) to (11). firstPass =(v x _firstPass,v y_firstPass). Similar to the current BDOF, the SAD or SATD between Pred0_sb8 and Pred1_sb8 may be calculated and compared with a threshold, where Pred0_sb8 is the predicted subblock of the current 8×8 subblock from reference image list 0, which is a subblock of Pred0_sb16ext, and Pred1_sb8 is the predicted subblock of the current 8×8 subblock from reference image list 1, which is a subblock of Pred1_sb16ext. If the SAD is less than the threshold, the refinement process for this 8×8 subblock is skipped. For example, the threshold may be set to k×sbWidth×sbHeight, where sbWidth and sbHeight are the width and height of the predicted subblock used for SAD or SATD calculation, and k is a positive factor. Figure 12B As shown, for an 8×8 sub-block, the pixels in the extended 14×14 area are required to derive the motion improved MV by formulas (1) to (4) or formulas (7) to (11). firstPass After deriving the first round of motion refinement based on optical flow, the motion vector of each 8×8 sub-block can be derived as: Among them, MV0 OFpass1 and MV1 OFpass1 MV0 is the motion vector improved by the first round of optical flow-based improvement for reference picture list 0 and reference picture list 1, respectively. pass2 and MV1 pass2 is the motion vector modified by the second bypass of multiple rounds of DMVR, MV firstPass is the motion refinement derived in the first round of optical flow-based motion refinement. sbIdx3 is the index of the sub-block used for the first round of optical flow-based motion refinement. sbIdx2 is the index of the sub-block used for the second round of multi-round DMVR.

[0143] After the first round of motion vector improvement based on optical flow, for each 8×8 sub-block, based on the MV0 obtained in the first round of motion improvement based on optical flow OFpass1 and MV1 OFpass1 , two prediction blocks Pred0_sb8ext and Pred1_sb8ext are obtained by inter-frame prediction, wherein Pred0_sb8ext is obtained by using MV0 OFpass1 The prediction block is obtained from the reference picture list 0, and Pred1_sb8ext is obtained by using MV1 OFpass1The prediction block is obtained from the reference image list 1. In the inter-frame prediction, an interpolation filter can be applied to generate the predicted pixel value at the sub-pixel position. The 8-tap, 12-tap or other interpolation filter can be used. Since the gradient is required in the optical flow formula, the prediction block is expanded. In this embodiment, Figure 13A As shown in FIG. 1 , 3 pixel rows and 3 pixel columns are extended on the four boundaries of the 8×8 sub-block. In some other embodiments, in order to reduce bandwidth and interpolation calculations, the extended pixels are filled from the boundary pixels within the 8×8 sub-block instead of actually interpolating them (e.g., Figure 5 In some embodiments, in order to save computation, the modified motion vectors of two adjacent 8×8 sub-blocks are checked. If two adjacent 8×8 sub-blocks have the same modified motion vector MV0 OFpass1 and MV1 OFpass1 , the two adjacent 8×8 blocks can be merged into 16×8 or 8×16 sub-blocks. Then, the implemented motion vector of the merged sub-block is continuously compared with the improved motion vector of its adjacent 8×8 sub-block. If the motion vectors are still the same, the two sub-blocks are further merged. The merging process is repeated to find the largest possible sub-block. Then, the inter-frame prediction is performed on the merged sub-block. Assuming that the second round of optical flow-based motion improvement is performed at the L×L sub-block level, in some other embodiments, the inter-frame prediction can also be performed separately on each L×L sub-block within the 8×8 sub-block. This will not affect the result.

[0144] After obtaining the two prediction blocks, the optical flow-based motion vector improvement method can be applied to each L×L sub-block in the 8×8 sub-block, where L is a positive integer equal to or less than 8. For example, the motion improvement MV of the second round of optical flow-based motion improvement is derived using formulas (1) to (4) or formulas (7) to (11). secondPass =(v x _secondPass,v y_secondPass). Similar to the current BDOF, the SAD or SATD between Pred0_sbL and Pred1_sbL may be calculated and compared with a threshold, where Pred0_sbL is the predicted subblock of the current L×L subblock from reference image list 0, which is a subblock of Pred0_sb8ext, and Pred1_sb4 is the predicted subblock of the current L×L subblock from reference image list 1, which is a subblock of Pred1_sb8ext. If the SAD is less than the threshold, the improvement process of this L×L subblock is skipped. For example, the threshold may be set to k×sbWidth×sbHeight, where sbWidth and sbHeight are the width and height of the predicted subblock used for SAD or SATD calculation, and k is a positive factor. As Figure 13B As shown, for an L×L sub-block, the pixels in the extended (L+6)×(L+6) area are required to derive the motion improved MV by formulas (1) to (4) or formulas (7) to (11). secondPass After deriving the second round of motion refinement based on optical flow, the motion vector of each L×L sub-block can be derived as: Among them, MV0 OFpass2 and MV1 OFpass2 MV0 is the motion vector improved by the second round of optical flow based improvement for reference picture list 0 and reference picture list 1 respectively. OFpass1 and MV1 OFpass1 are the motion vectors improved by the first round of optical flow-based improvement for reference picture list 0 and reference picture list 1, respectively. secondPass is the motion refinement derived in the second round of optical flow-based motion refinement, sbIdx4 is the index of the sub-block used for the second round of optical flow-based motion refinement, and sbIdx3 is the index of the sub-block used for the first round of optical flow-based motion refinement.

[0145] The derived MV0 OFpass2 and MV0 OFpass1 is the final motion vector of the current L×L sub-block. After that, existing motion compensation can be performed. Sub-block-based BDOF or sample-based BDOF can also be used in motion compensation.

[0146] In the above embodiment, the proposed multi-round optical flow-based motion improvement follows the first two rounds of multi-round DMVR. By doing so, the multi-round DMVR is extended to four rounds. However, the proposed method is not necessarily combined with multi-round DMVR. It can be used independently of DMVR. For a bidirectionally predicted CU, if some conditions are met, the proposed multi-round optical flow-based motion improvement can be performed to improve the motion of the CU. Therefore, the input of the proposed multi-round optical flow-based motion improvement can be the motion vector derived in the merge mode, which is inherited from the previous coding block or decoded from the bitstream. Moreover, the output of the proposed multi-round optical flow-based motion improvement can be further improved by other coding tools before motion compensation. The conditions for multi-round optical flow-based motion improvement may include but are not limited to the following situations: - The CU is encoded using "true" bi-prediction mode, i.e. one of the two reference pictures precedes the current picture in display order and the other follows the current picture in display order - The distances from the two reference images to the current image (i.e., POC differences) are the same - Both reference images are short-term reference images - The CU is not encoded using affine mode or SbTMVP merge mode, local illumination compensation prediction mode, or overlapped block motion compensation -CU has more than 64 luma samples -CU height and CU width are both greater than or equal to 8 luma samples - Bidirectional prediction with CU level weight (BCW) weight index indicating equal weight - Weighted Prediction (WP) is not enabled for the current CU - Inter-frame and intra-frame joint prediction (CIIP) mode is not used for the current CU - Do not use symmetric motion vector differences -Do not encode the CU in merge mode with motion vector difference

[0147] In some embodiments, a difference between two prediction blocks of the current CU (one from reference picture list 0 and the other from reference picture list 1) is calculated, and based on the difference, optical flow-based refinement is skipped. The difference may be the sum of absolute differences (SAD) or the sum of absolute differences (SATD) of the two prediction blocks.

[0148] In one example, the number of refinement rounds depends on the difference between the two prediction blocks. If the difference is less than or equal to a threshold, only one round of optical flow-based refinement is performed (skipping the second round); otherwise, two rounds of optical flow-based refinement are performed. Alternatively, if the difference is less than or equal to a first threshold, no optical flow-based refinement is performed; if the difference is greater than the first threshold but less than or equal to a second threshold, only one round of optical flow-based refinement is performed; and if the difference is greater than the second threshold, three rounds of optical flow-based refinement are performed.

[0149] In another example, skipping or not skipping the optical flow-based refinement depends on the difference between the two prediction blocks. First, the two prediction blocks are interpolated by using the motion vector before refinement (denoted as MV0), and the difference between the two prediction blocks is calculated, denoted as D0. If D0 is less than or equal to a first threshold, no optical flow-based refinement is performed; if D0 is greater than the threshold, the first round of optical flow-based refinement is performed, and the motion vector is refined, denoted as MV1. Then, two more prediction blocks are obtained by using the refined motion vector MV1, and the difference between the two prediction blocks is calculated, denoted as D1, and compared with a second threshold. If D1 is less than or equal to the second threshold, the refinement process is terminated; if D0 is greater than the threshold, the second round of optical flow-based refinement is continued, and the motion vector is refined again.

[0150] In yet another example, whether to perform optical flow-based improvement is decided based on the change in the difference between two prediction blocks. First, the two prediction blocks are interpolated by using the motion vector before improvement (denoted as MV0), and the difference between the two prediction blocks is calculated, denoted as D0. Then, the optical flow-based improvement is performed, and the motion vector is improved, denoted as MV1. Then, two other prediction blocks are obtained by using the improved motion vector MV1, and the difference between the two prediction blocks is calculated, denoted as D1. If D1 is greater than D0, the result of the optical flow-based improvement is restored. That is, the improved motion vector MV1 is replaced by the original motion vector MV0; if D1 is less than or equal to D0, the result of the optical flow-based improvement is retained, and the following process is performed based on the improved motion vector MV1. In some other examples, the change in the difference is used to decide whether to perform another round of optical flow-based improvement. If D1 is greater than D0, the refinement process is terminated and the refined motion vector MV1 is output to the next stage. If D1 is less than or equal to D0, another round of optical flow-based refinement is performed based on MV1 to obtain another refined motion vector MV2. Therefore, the number of rounds of optical flow-based refinement is based on the difference between the difference values ​​between the two prediction blocks before and after refinement.

[0151] Variations of the above embodiments can be developed to implement the disclosed optical flow-based motion refinement method. In some embodiments, there may be more than two rounds of optical flow-based motion refinement for a CU. For example, the first round is performed at the 8×8 sub-block level, the second round is performed at the 4×4 sub-block level, and the third round is performed at the 2×2 sub-block level. As another example, the first round is performed at the 8×8 sub-block level, the second round is performed at the 8×8 sub-block level, and the third round is also performed at the 8×8 sub-block level.

[0152] In some embodiments, the number of rounds of optical flow-based motion refinement depends on the CU size, the sequence resolution, or the quantization parameter. For example, there are fewer rounds for large CUs, while there are more rounds for small CUs, because large CUs typically have smoother content that can be refined on a large matrix and may not require multiple rounds of optical flow-based motion refinement. For example, there is only one round for CUs with an area greater than or equal to 4096; two rounds for CUs with an area less than 4096 but greater than 256; and three rounds for CUs with an area less than 256. For another example, there are fewer rounds for CUs encoded with large quantization parameters, while there are more rounds for CUs encoded with small quantization parameters, because larger quantization parameters result in more distortion and therefore require more rounds of refinement. For yet another example, there are fewer rounds for large video sequences, and more rounds for small video sequences, because large video sequences typically have more large CUs that do not require more rounds of refinement, while small video sequences typically have more small CUs that require more rounds of refinement.

[0153] In some embodiments, the sub-block size used for optical flow-based motion improvement depends on the CU size, quantization parameter or video sequence resolution. For example, for a large CU, the optical flow-based motion improvement is performed at the large sub-block level, while for a small CU, the optical flow-based motion improvement is performed at the small sub-block level. For example, for a CU larger than 256, the optical flow-based improvement is performed at the 8×8 sub-block level, while for a CU smaller than or equal to 256, the optical flow-based improvement is performed at the 4×4 sub-block level. For another example, for a CU encoded with a large number of quantization parameters, the optical flow-based motion improvement is performed at the large sub-block level, while for a CU encoded with a small quantization parameter, the optical flow-based motion improvement is performed at the small sub-block level. For yet another example, for a large video sequence, the optical flow-based motion improvement is performed at the large sub-block level, while for a small video sequence, the optical flow-based motion improvement is performed at the small sub-block level.

[0154] Furthermore, the adaptive sub-block size for optical flow-based motion improvement can be combined with multiple rounds of optical flow-based motion improvement. For example, for a CU with an area greater than 1024, the first round of optical flow-based motion improvement is performed at the 16×16 sub-block level, and the second round is performed at the 8×8 sub-block level; for a CU with an area less than or equal to 1024, the first round of optical flow-based motion improvement is performed at the 8×8 sub-block level, and the second round is performed at the 4×4 sub-block level. For another example, for the CU with an area greater than or equal to 1024, two rounds are performed at the 8×8 sub-block level; for a CU with an area less than 1024 but greater than or equal to 512, the first round is performed at the 8×8 sub-block level, and the second round is performed at the 4×4 sub-block level; and for a CU with an area less than 512, two rounds are performed at the 4×4 sub-block level. For another example, for a CU encoded with a quantization parameter greater than or equal to 32, the first round of optical flow-based motion improvement is performed at the 16×16 sub-block level, and the second round is performed at the 8×8 sub-block level; for a CU encoded with a quantization parameter less than 32, the first round of optical flow-based motion improvement is performed at the 8×8 sub-block level, and the second round is performed at the 4×4 sub-block level. For another example, for a video sequence with a resolution greater than 1920×1080, the first round of optical flow-based motion improvement is performed at the 16×16 sub-block level, and the second round is performed at the 8×8 sub-block level; for a video sequence with a resolution less than 1920×1080, the first round of optical flow-based motion improvement is performed at the 8×8 sub-block level, and the second round is performed at the 4×4 sub-block level.

[0155] In addition, the adaptive sub-block size improvement of optical flow-based motion improvement can be combined with an adaptive number of rounds of optical flow-based motion improvement. For example, for a CU with an area greater than 1024, there is only one round of optical flow-based motion improvement, which is performed at the 16×16 sub-block level; for a CU with an area less than or equal to 1024, there are two rounds of optical flow-based motion improvement, wherein the first round is performed at the 8×8 sub-block level and the second round is performed at the 4×4 sub-block level. As another example, for a CU encoded with a quantization parameter greater than or equal to 32, there are two rounds of optical flow-based motion improvement, wherein the first round is performed at the 16×16 sub-block level and the second round is performed at the 8×8 sub-block level; for a CU encoded with a quantization parameter less than 32, there is only one round of optical flow-based motion improvement, which is performed at the 8×8 sub-block level. As another example, for video sequences with a resolution greater than 1920×1080, there is only one round of optical flow-based motion refinement, which is performed at the 16×16 sub-block level; for video sequences with a resolution less than 1920×1080, there are two rounds of optical flow-based motion refinement, where the first round is performed at the 8×8 sub-block level and the second round is performed at the 4×4 sub-block level.

[0156] The embodiments described in this disclosure can be freely combined.

[0157] In some embodiments, a non-transitory computer-readable storage medium storing a bitstream is further provided. The bitstream can be encoded and decoded according to the disclosed optical flow-based motion improvement method.

[0158] In some embodiments, a non-transitory computer-readable storage medium comprising instructions is also provided, and the instructions can be executed by a device (e.g., the disclosed encoder and decoder) for performing the above method. Common forms of non-transitory media include, for example, floppy disks, flexible magnetic disks, hard disks, solid-state drives, tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM or any other flash memory, NVRAM, caches, registers, any other memory chips or cassettes, and networked versions thereof. The device may include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memories.

[0159] The embodiments may be further described using the following terms: 1. A video processing method, comprising: Dividing the coding block into a first group of sub-blocks and a second group of sub-blocks; performing a first round of optical flow-based motion vector refinement on the first group of sub-blocks; and A second round of optical flow-based motion vector refinement is performed on the second group of sub-blocks. 2. The method of clause 1, wherein the size of the first group of sub-blocks is different from the size of the second group of sub-blocks. 3. The method according to clause 2, further comprising: Dividing the coding block into a third group of sub-blocks; and A third round of optical flow-based motion vector refinement is performed on the third group of sub-blocks. 4. The method according to clause 2, wherein: Each of the first group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; Each of the second group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; and Each of the third group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16. 5. The method of clause 2, wherein the optical flow based motion vector refinement is bidirectional. 6. The method according to clause 2, further comprising: Before performing the first, second and third rounds of optical flow-based motion vector improvement, one or more rounds of decoding-side motion vector improvement (DMVR) are performed on the coding block. 7. The method of clause 6, wherein performing the one or more rounds of DMVR on the coded block comprises: performing a first round of bilateral matching motion vector refinement on the coding block; Dividing the coding block into a fourth group of sub-blocks; and A second round of bilateral matching motion vector refinement is performed on the fourth group of sub-blocks. 8. The method of clause 2, wherein the first, second, and third rounds of optical flow-based motion vector refinement belong to multiple rounds of optical flow-based motion vector refinement performed on the coding block, and the method further comprises: The number of rounds is determined based on the size of the coding block. 9. The method according to clause 2, further comprising: A subblock size of each of the first group of subblocks, the second group of subblocks, or the third group of subblocks is determined based on whether a predetermined condition is satisfied. 10. The method of clause 9, wherein the predetermined condition comprises at least one of the following: Encoding the coding block using a bidirectional prediction mode; The coding block has a size exceeding a predetermined threshold; The coding block has a size smaller than a predetermined threshold; disabling weighted prediction for the coding block; disabling CIIP for the coding block; enabling local brightness compensation for the coding block; applying sub-block motion compensation to the coding block; Not using symmetric motion vector differences; or The coding block is not encoded as merge mode with motion vector difference. 11. The method of clause 1, further comprising: generating inter-frame prediction samples based on the modified motion vector; and A bitstream is encoded based on the inter-frame prediction samples. 12. The method of clause 1, further comprising: decoding a bitstream associated with the video; and The coded block is reconstructed. 13. An apparatus comprising: a memory storing computer instructions; and one or more processors configured to execute the computer instructions, wherein the execution of the computer instructions causes the apparatus to perform operations comprising: Dividing the coding block into a first group of sub-blocks and a second group of sub-blocks; performing a first round of optical flow-based motion vector refinement on the first group of sub-blocks; and A second round of optical flow-based motion vector refinement is performed on the second group of sub-blocks. 14. The apparatus of clause 13, wherein the size of the first set of sub-blocks is different from the size of the second set of sub-blocks. 15. The apparatus of clause 14, wherein the execution of the computer instructions causes the apparatus to further perform: Dividing the coding block into a third group of sub-blocks; and A third round of optical flow-based motion vector refinement is performed on the third group of sub-blocks. 16. The apparatus of clause 14, wherein: Each of the first group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; Each of the second group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; and Each of the third group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16. 17. The apparatus of clause 14, wherein the optical flow-based motion vector refinement is bidirectional. 18. The apparatus of clause 14, wherein execution of the computer instructions causes the apparatus to further: Before performing the first, second and third rounds of optical flow based motion vector refinement, one or more rounds of decoding-side motion vector refinement (DMVR) are performed on the coding block. 19. The apparatus of clause 18, wherein, when performing the one or more rounds of DMVR on the encoded block, execution of the computer instructions causes the apparatus to further perform: performing a first round of bilateral matching motion vector refinement on the coding block; Dividing the coding block into a fourth group of sub-blocks; and A second round of bilateral matching motion vector refinement is performed on the fourth group of sub-blocks. 20. The apparatus of clause 14, wherein the first, second, and third rounds of optical flow-based motion vector refinement belong to multiple rounds of optical flow-based motion vector refinement performed on the coding block, and execution of the computer instructions causes the apparatus to further perform: The number of rounds is determined based on the size of the coding block. 21. The apparatus of clause 14, wherein execution of the computer instructions causes the apparatus to further: A subblock size of each of the first group of subblocks, the second group of subblocks, or the third group of subblocks is determined based on whether a predetermined condition is satisfied. 22. The apparatus of clause 21, wherein the predetermined condition comprises at least one of the following: Encoding the coding block using a bidirectional prediction mode; The coding block has a size exceeding a predetermined threshold; The coding block has a size smaller than a predetermined threshold; disabling weighted prediction for the coding block; disabling CIIP for the coding block; enabling local brightness compensation for the coding block; applying sub-block motion compensation to the coding block; Not using symmetric motion vector differences; or The coding block is not encoded as merge mode with motion vector difference. 23. The apparatus of clause 13, wherein execution of the computer instructions causes the apparatus to further: generating inter-frame prediction samples based on the modified motion vector; and A bitstream is encoded based on the inter-frame prediction samples. 24. The apparatus of clause 13, wherein the execution of the computer instructions causes the apparatus to further: decoding a bitstream associated with the video; and The coded block is reconstructed. 25. A non-transitory computer-readable storage medium storing a bitstream of a video for processing according to a method comprising: Dividing the coding block into a first group of sub-blocks and a second group of sub-blocks; performing a first round of optical flow-based motion vector refinement on the first group of sub-blocks; and A second round of optical flow-based motion vector refinement is performed on the second group of sub-blocks. 26. The non-transitory computer-readable storage medium of clause 25, wherein a size of the first group of sub-blocks is different than a size of the second group of sub-blocks. 27. The non-transitory computer-readable storage medium of clause 26, wherein the method further comprises: Dividing the coding block into a third group of sub-blocks; and A third round of optical flow-based motion vector refinement is performed on the third group of sub-blocks. 28. The non-transitory computer-readable storage medium of clause 26, wherein: Each of the first group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; Each of the second group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; and Each of the third group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16. 29. The non-transitory computer-readable storage medium of clause 26, wherein the optical flow-based motion vector refinement is bidirectional. 30. The method according to clause 2, further comprising: Before performing the first, second and third rounds of optical flow-based motion vector improvement, one or more rounds of decoding-side motion vector improvement (DMVR) are performed on the coding block. 31. The non-transitory computer-readable storage medium of clause 30, wherein performing the one or more rounds of DMVR on the coded block comprises: performing a first round of bilateral matching motion vector refinement on the coding block; Dividing the coding block into a fourth group of sub-blocks; and A second round of bilateral matching motion vector refinement is performed on the fourth group of sub-blocks. 32. The non-transitory computer-readable storage medium of clause 26, wherein the first, second, and third rounds of optical flow-based motion vector refinement belong to multiple rounds of optical flow-based motion vector refinement performed on the coding block, and the method further comprises: The number of passes is determined based on a size of the coding block. 33. The non-transitory computer-readable storage medium of clause 26, wherein the method further comprises: A subblock size of each of the first group of subblocks, the second group of subblocks, or the third group of subblocks is determined based on whether a predetermined condition is satisfied. 34. The non-transitory computer-readable storage medium of clause 33, wherein the predetermined condition comprises at least one of the following: Encoding the coding block using a bidirectional prediction mode; The coding block has a size exceeding a predetermined threshold; The coding block has a size smaller than a predetermined threshold; disabling weighted prediction for the coding block; disabling CIIP for the coding block; enabling local brightness compensation for the coding block; applying sub-block motion compensation to the coding block; Not using symmetric motion vector differences; or The coding block is not encoded as merge mode with motion vector difference. 35. The non-transitory computer-readable storage medium of clause 25, wherein the method further comprises: generating inter-frame prediction samples based on the modified motion vector; and A bitstream is encoded based on the inter-frame prediction samples. 36. The non-transitory computer-readable storage medium of clause 25, wherein the method further comprises: decoding a bitstream associated with the video; and The coded block is reconstructed.

[0160] It should be noted that relational terms such as "first," "second," and the like in this document are used only to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or sequence between these entities or operations. In addition, the words "comprising," "having," "containing," and "including," and other similar forms are intended to be synonymous and open-ended, in that one or more items following any of these words are not intended to be an exhaustive list of such one or more items, nor are they intended to be limited to the listed one or more items.

[0161] As used herein, unless otherwise specifically stated, the term "or" encompasses all possible combinations unless not feasible. For example, if a database is specified to include either A or B, then unless otherwise specifically stated or not feasible, the database may include either A, or B, or A and B. As a second example, if a database is specified to include either A, B, or C, then unless otherwise specifically stated or not feasible, the database may include either A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0162] It should be understood that the above embodiments can be implemented by hardware, or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-mentioned computer-readable medium. When the software is executed by a processor, the software can execute the disclosed method. The computing unit and other functional units described in this disclosure can be implemented by hardware, or software, or a combination of hardware and software. Those of ordinary skill in the art will also understand that multiple modules / units in the modules / units described above can be combined into one module / unit, and each module / unit described above can be further divided into multiple sub-modules / sub-units.

[0163] In the foregoing description, embodiments have been described with reference to many specific details, which may vary depending on the implementation. Certain adjustments and modifications may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art by considering the description and practice of the invention disclosed herein. The description and examples are intended to be exemplary only, with the true scope and spirit of the invention being indicated by the appended claims. The sequence of steps shown in the figures is also intended to be for illustrative purposes only and is not intended to be limited to any particular sequence of steps. Therefore, it will be understood by those skilled in the art that these steps may be performed in different orders while implementing the same method.

[0164] In the drawings and the specification, exemplary embodiments have been disclosed. However, many variations and modifications may be made to these embodiments. Therefore, although specific terms are employed, these terms are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. A video processing method, comprising: Dividing the coding block into a first group of sub-blocks and a second group of sub-blocks; performing a first round of optical flow-based motion vector refinement on the first group of sub-blocks; as well as A second round of optical flow-based motion vector refinement is performed on the second group of sub-blocks.

2. The method according to claim 1, wherein The size of the first group of sub-blocks is different from the size of the second group of sub-blocks.

3. The method according to claim 2, further comprising: Dividing the coding block into a third group of sub-blocks; as well as A third round of optical flow-based motion vector refinement is performed on the third group of sub-blocks.

4. The method according to claim 2, wherein: Each of the first group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; Each of the second group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; and Each of the third group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16.

5. The method according to claim 2, wherein: The optical flow based motion vector refinement is bidirectional.

6. The method according to claim 2, further comprising: Before performing the first, second and third rounds of optical flow-based motion vector refinement, one or more rounds of decoding-side motion vector refinement (DMVR) are performed on the coding block.

7. The method according to claim 6, wherein: Performing the one or more rounds of DMVR on the coding block includes: performing a first round of bilateral matching motion vector refinement on the coding block; Dividing the coding block into a fourth group of sub-blocks; and A second round of bilateral matching motion vector refinement is performed on the fourth group of sub-blocks.

8. The method according to claim 2, wherein: The first round, the second round, and the third round of optical flow-based motion vector improvement belong to multiple rounds of optical flow-based motion vector improvement performed on the coding block, and the method further includes: The number of rounds is determined based on the size of the coding block.

9. The method according to claim 2, further comprising: A subblock size of each of the first group of subblocks, the second group of subblocks, or the third group of subblocks is determined based on whether a predetermined condition is satisfied.

10. The method according to claim 9, wherein: The predetermined condition includes at least one of the following: Encoding the coding block using a bidirectional prediction mode; The coding block has a size exceeding a predetermined threshold; The coding block has a size smaller than a predetermined threshold; disabling weighted prediction for the coding block; disabling CIIP for the coding block; enabling local brightness compensation for the coding block; applying sub-block motion compensation to the coding block; Symmetric motion vector differences are not used; or The coding block is not encoded as merge mode with motion vector difference.

11. The method according to claim 1 , further comprising: generating inter-frame prediction samples based on the modified motion vector; as well as A bitstream is encoded based on the inter-frame prediction samples.

12. The method according to claim 1, further comprising: decoding a bitstream associated with the video; and The coded block is reconstructed.

13. An apparatus comprising: a memory storing computer instructions; as well as one or more processors configured to execute the computer instructions, wherein the execution of the computer instructions causes the apparatus to perform operations comprising: Dividing the coding block into a first group of sub-blocks and a second group of sub-blocks; performing a first round of optical flow-based motion vector refinement on the first group of sub-blocks; and A second round of optical flow-based motion vector refinement is performed on the second group of sub-blocks.

14. The device according to claim 13, wherein The size of the first group of sub-blocks is different from the size of the second group of sub-blocks.

15. The device according to claim 14, wherein Execution of the computer instructions causes the apparatus to further: Dividing the coding block into a third group of sub-blocks; and A third round of optical flow-based motion vector refinement is performed on the third group of sub-blocks.

16. The apparatus according to claim 14, wherein: Each of the first group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; Each of the second group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16; and Each of the third group of sub-blocks has a size of 2×2, 4×4, 8×8, or 16×16.

17. The device according to claim 14, wherein The optical flow-based motion vector is bidirectional.

18. The device according to claim 14, wherein Execution of the computer instructions causes the apparatus to further: Before performing the first, second and third rounds of optical flow-based motion vector improvement, one or more rounds of decoding-side motion vector improvement (DMVR) are performed on the coding block.

19. The device according to claim 18, wherein While performing the one or more rounds of DMVR on the encoded block, the execution of the computer instructions causes the apparatus to further perform: performing a first round of bilateral matching motion vector refinement on the coding block; Dividing the coding block into a fourth group of sub-blocks; and A second round of bilateral matching motion vector refinement is performed on the fourth group of sub-blocks.

20. A non-transitory computer-readable storage medium having stored thereon a video bitstream for processing according to a method comprising: Dividing the coding block into a first group of sub-blocks and a second group of sub-blocks; performing a first round of optical flow-based motion vector refinement on the first group of sub-blocks; as well as A second round of optical flow-based motion vector refinement is performed on the second group of sub-blocks.

Images