OPTICAL FLOW-BASED MOTION REFINEMENT

Description

1 / 77 OPTICAL FLOW-BASED MOTION REFINEMENT CROSS-REFERENCE TO RELATED ORDERS

[0001] The present invention claims the benefit of priority of: U.S. Provisional Application No. 63 / 495,768, filed April 12, 2023, U.S. Provisional Application No. 63 / 508,010, filed June 14, 2023, and U.S. Application No. 18 / 628,086, filed April 5, 2024, the contents of which are incorporated by reference herein in their entirety. TECHNICAL FIELD

[0002] The present invention relates generally to video processing and, more specifically, to methods and apparatus for motion refinement based on optical flow. BACKGROUND

[0003] A video is a collection of still images (or “frames”) that capture visual information. To reduce storage memory and transmission bandwidth, a video can be compressed before storage or transmission, and decompressed before display. The compression process is generally referred to as encoding, and the decompression process is generally referred to as decoding. There are several video encoding formats that use standardized video encoding technologies, generally based on prediction, transformation, quantization, entropy coding, and loop filtering. Video encoding standards, such as the High Efficiency Video Coding Standard (HEVC / H.265), Versatile Video Coding Standard (VVC / H.266), and AVS standards, which specify specific video encoding formats, are developed by standardization organizations.With increasingly advanced video encoding technologies being adopted in video standards, the encoding efficiency of the new video encoding standards is becoming ever greater. Petition 870250092849, dated 10 / 10 / 2025, page 9 / 119 2 / 77 SUMMARY OF DESCRIPTION

[0004] The embodiments of the present invention provide methods and apparatus for motion refinement based on optical flow.

[0005] According to some exemplary embodiments, a video processing method is provided including: dividing a coding block into a first set of sub-blocks and a second set of sub-blocks; performing a first optical flow-based motion vector refinement step on the first set of sub-blocks; and performing a second optical flow-based motion vector refinement step on the second set of sub-blocks.

[0006] According to some exemplary embodiments, an apparatus is provided including: a memory that stores computer instructions; and one or more processors configured to execute the computer instructions. The execution of the computer instruction causes the apparatus to perform operations including: dividing a coding block into a first set of sub-blocks and a second set of sub-blocks; performing a first optical flow-based motion vector refinement step on the first set of sub-blocks; and performing a second optical flow-based motion vector refinement step on the second set of sub-blocks.

[0007] According to some exemplary embodiments, a non-transient computer-readable storage medium is provided that stores a bitstream of a video. The bitstream is intended for processing according to a method, including: dividing a coding block into a first set of sub-blocks and a second set of sub-blocks; performing a first optical flow-based motion vector refinement step on the first set of sub-blocks; and performing a second refinement step. Petition 870250092849, dated 10 / 10 / 2025, page 10 / 119 3 / 77 of optical flow-based motion vector in the second set of sub-blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The embodiments and various aspects of the present invention are illustrated in the following detailed description and in the accompanying figures. Several features in the figures are not shown to scale.

[0009] FIG. 1 is a schematic diagram illustrating structures of an exemplary video sequence, according to some embodiments of the present invention.

[0010] FIG. 2A is a schematic diagram illustrating an exemplary encoding process of a hybrid video encoding system, according to some embodiments of the present invention.

[0011] FIG. 2B is a schematic diagram illustrating another exemplary encoding process of a hybrid video encoding system, according to some embodiments of the present invention.

[0012] FIG. 3A is a schematic diagram illustrating an exemplary decoding process of a hybrid video encoding system, according to some embodiments of the present invention.

[0013] FIG. 3B is a schematic diagram illustrating another exemplary decoding process of a hybrid video encoding system, according to some embodiments of the present invention.

[0014] FIG. 4 is a block diagram of an exemplary apparatus for encoding or decoding video, according to some embodiments of the present invention.

[0015] FIG. 5 is a schematic diagram illustrating a region Petition 870250092849, dated 10 / 10 / 2025, page 11 / 119 4 / 77 of extended coding unit (CU) used in bidirectional optical flow (BDOF), according to some embodiments of the present invention.

[0016] FIG. 6 is a schematic diagram illustrating a motion vector refinement process aimed at decoding (DMVR), according to some embodiments of the present invention.

[0017] FIG. 7 is a schematic diagram illustrating a 3*3 square search pattern, according to some embodiments of the present invention.

[0018] FIG. 8 is a schematic diagram illustrating diamond regions in a search area for refined motion vectors (MV), according to some embodiments of the present invention.

[0019] FIG. 9 illustrates a flowchart of an exemplary method for performing motion vector refinement and motion compensation, according to some embodiments of the present invention.

[0020] FIG. 10 illustrates a flowchart of an exemplary method for performing motion vector refinement and motion compensation, according to some embodiments of the present invention.

[0021] FIG. 11 is a flowchart of another exemplary method for performing motion vector refinement and motion compensation, according to some embodiments of the present invention.

[0022] FIGS. 12A and 12B are schematic diagrams illustrating a process for splitting a predicted block using an extended predicted block, according to some embodiments of the present invention.

[0023] FIGS. 13A and 13B are schematic diagrams illustrating a process for splitting a predicted block using an extended predicted block, according to some embodiments of the present invention. Petition 870250092849, dated 10 / 10 / 2025, page 12 / 119 5 / 77 DETAILED DESCRIPTION

[0024] Detailed reference will now be made to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent identical or similar elements, except where otherwise represented. The implementations defined in the following description of the exemplary embodiments do not represent all implementations consistent with the invention. Instead, they are only examples of apparatus and methods consistent with aspects related to the invention as cited in the accompanying claims. The specific aspects of the present invention are described in more detail below. The terms and definitions provided in this document prevail if they conflict with terms or definitions incorporated by reference.

[0025] The embodiments provided by the present invention are directed to encoding and decoding video information and, more specifically, to methods and systems for performing optical flow-based motion vector refinement. The disclosed methods perform multiple steps of optical flow-based motion vector refinement and adaptively adjust the subblock mesh size, thus leading to more stable results in optical flow calculation.

[0026] To reduce the storage space and transmission bandwidth required by such applications, video can be compressed before storage and transmission and decompressed before display. Compression and decompression can be implemented by software running on a processor (e.g., a processor in a generic computer) or specialized hardware. The module for compression is generally referred to as an “encoder”. Petition 870250092849, dated 10 / 10 / 2025, p. 13 / 119 6 / 77 pain”, and the module for decompression is generally referred to as a “decoder”. The encoder and decoder may be collectively referred to as a “codec”. The encoder and decoder may be implemented as any variety of hardware, software, or suitable combination thereof. For example, the hardware implementation of the encoder and decoder may include circuits such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, or any combination thereof. The software implementation of the encoder and decoder may include program codes, computer executable instructions, firmware, or any suitable computer-implemented algorithm or process fixed on a computer-readable medium.Video compression and decompression can be implemented by various algorithms or standards, such as the MPEG-1, MPEG-2, MPEG-4, H.26x, AVS series, or similar. In some applications, the codec may decompress the video from a first encoding standard and then compress the decompressed video again using a second encoding standard, in which case the codec may be referred to as a "transcoder".

[0027] The video encoding process can identify and retain useful information that can be used to reconstruct an image and discard irrelevant information for reconstruction. If the discarded and irrelevant information cannot be fully reconstructed, the aforementioned encoding process can be referred to as “lossy”. Otherwise, the same process can be referred to as “lossless”. Most encoding processes occur with data loss, which is a trade-off to reduce the required storage space and transmission bandwidth. Petition 870250092849, dated 10 / 10 / 2025, page 14 / 119 7 / 77

[0028] Useful information from an image being encoded (referred to as a “current image”) includes changes with respect to a reference image (e.g., a previously encoded and reconstructed image). Such changes may include changes in position, brightness, or color of the pixels, with position changes being the most relevant. Changes in the position of a group of pixels representing an object may reflect the object's movement between the reference image and the current image.

[0029] An encoded image without reference to another image (i.e., it is its own reference image) is referred to as an “I-picture”. An image is referred to as a “P-picture” if some or all of the blocks (e.g., blocks that generally refer to video image positions) in the image are predicted using intraprediction or interprediction with a reference image (e.g., uniprediction). An image is referred to as a “B-picture” if at least one block in it is predicted with two referenced images (e.g., bi-prediction).

[0030] FIGS. 1, 2A, 2B, 3A, 3B and 4 illustrate the general aspects of the video encoding / decoding apparatus and processes used in the disclosed embodiments. Specifically, FIG. 1 illustrates structures of an exemplary video sequence 100, according to some embodiments of the present invention. The video sequence 100 can be a live video or a video that has been captured and archived. The video 100 can be a real-life video, a computer-generated video (e.g., computer game video), or a combination thereof (e.g., a real-life video with augmented reality effects). The video sequence 100 can be introduced from a video capture device (e.g., a camera). Petition 870250092849, dated 10 / 10 / 2025, page 15 / 119 8 / 77 mera), a video file (for example, a video file stored on a video device) containing previously captured video, or a video transmission interface (for example, a video broadcast transceiver) for receiving video from a video content provider.

[0031] As shown in FIG. 1, the video sequence 100 may include a series of images arranged temporally along a timeline, including images 102, 104, 106, and 108. Images 102–106 are continuous, and there are more images between images 106 and 108. In FIG. 1, image 102 is an I-image, the reference image of image 102 itself. Image 104 is a P-image, the reference image of image 102, as indicated by the arrow. Image 106 is a B-image, the reference images of images 104 and 108, as indicated by the arrows. In some embodiments, the reference image of an image (e.g., image 104) may not be immediately preceding or following the image. For example, the reference image of image 104 may be an image preceding image 102.It should be noted that the reference images in Figures 102-106 are merely examples, and the present invention does not limit the embodiments of the reference images as exemplified in FIG. 1.

[0032] Typically, video codecs do not encode or decode an entire image at once due to the computational complexity of such tasks. Instead, they divide the image into basic segments and encode or decode the image segment by segment. These basic segments are referred to as basic processing units (“BPUs”) in the present invention. For example, structure 110 in FIG. 1 shows an exemplary structure of a sequence image of 100 videos (e.g., any of images 102-108). In structure 110, an image is divided into units Petition 870250092849, dated 10 / 10 / 2025, page 16 / 119 9 / 77 basic 4*4 processing units, whose boundaries are shown in dashed lines. In some embodiments, basic processing units may be referred to as “macroblocks” in some video coding standards (e.g., MPEG, H.261, H.263, or H.264 / AVC families), or as “coding tree units” (“CTUs”) in some other video coding standards (e.g., H.265 / HEVC, H.266 / VVC, or AVS). Basic processing units may have variable sizes in an image, such as 128*128, 64*64, 32*32, 16*16, 4*8, 16*32, or any arbitrary pixel shape and size. The sizes and shapes of the basic processing units may be selected for an image based on the balance of coding efficiency and levels of detail to be maintained in the basic processing unit.

[0033] Basic processing units can be logical units, which may include a group of different types of video data stored in computer memory (e.g., in a video frame buffer). For example, a basic processing unit for a color image may include a luma component (Y) representing achromatic brightness information, one or more chroma components (e.g., Cb and Cr) representing color information, and associated syntax elements, in which the luma and chroma components may have the same size as the basic processing unit. The luma and chroma components may be referred to as “coding tree blocks” (“CTBs”) in some video encoding standards (e.g., H.265 / HEVC, H.266 / VVC, or AVS). Any operation performed on a basic processing unit can be repeatedly performed on each of its luma and chroma components.

[0034] Video encoding has multiple stages of operation, examples of which are shown in FIGS. 2A-2B and FIGS. 3A-3B. Petition 870250092849, dated 10 / 10 / 2025, page 17 / 119 10 / 77 For each stage, the size of the basic processing units may still be too large for processing; thus, they may be further divided into segments referred to as “basic processing subunits” in the present invention. In some embodiments, the basic processing subunits may be referred to as “blocks” in some video coding standards (e.g., MPEG family, H.261, H.263, H.264 / AVC, or AVS), or as “coding units” (“CUs”) in some other video coding standards (e.g., H.265 / HEVC, H.266 / VVC, or AVS). A basic processing subunit may be the same size as or smaller than the basic processing unit.Similar to basic processing units, basic processing subunits are also logical units, which may include a group of different types of video data (e.g., Y, Cb, Cr, and associated syntax elements) stored in computer memory (e.g., in a video frame buffer). Any operation performed on a basic processing subunit can be repeatedly performed on each of its luma and chroma components. It should be noted that each division can be further divided into additional levels depending on processing needs. It should also be noted that different stages may divide the basic processing units using different schemes.

[0035] For example, in a mode decision stage (an example of which is shown in FIG. 2B), the encoder can decide which prediction mode (e.g., intra-image prediction or inter-image prediction) to use for a basic processing unit, which may be too large to make such a decision. The encoder can divide the basic processing unit into multiple basic processing subunits (e.g., CUs as in H.265 / HEVC, H.266 / VVC, or AVS) and decide on a prediction type for each basic subunit. Petition 870250092849, dated 10 / 10 / 2025, page 18 / 119 11 / 77 individual processing.

[0036] For another example, in a prediction stage (an example of which is shown in FIGS. 2A-2B), the encoder can perform prediction operations at the level of basic processing subunits (e.g., CUs). However, in some cases, a basic processing subunit may still be too large to process. The encoder can further divide the basic processing subunit into smaller segments (e.g., referred to as “prediction blocks” or “PBs” in H.265 / HEVC, H.266 / VVC, or AVS), at the level at which prediction operations can be performed.

[0037] For another example, in a transform stage (an example of which is shown in FIGS. 2A-2B), the encoder can perform a transform operation for residual basic processing subunits (e.g., CUs). However, in some cases, a basic processing subunit may still be too large to process. The encoder can further divide the basic processing subunit into smaller segments (e.g., referred to as “transform blocks” or “TBs” in H.265 / HEVC, H.266 / VVC, or AVS), at the level at which the transform operation can be performed. It should be noted that the division schemes of the same basic processing subunit may be different in the prediction stage and the transform stage. For example, in H.265 / HEVC, H.266 / VVC, or AVS, the prediction blocks and the transform blocks of the same CU may have different sizes and numbers.

[0038] In structure 110 of FIG. 1, the basic processing unit 112 is further divided into 3x3 basic processing subunits, whose boundaries are shown in dashed lines. Different basic processing units of the same image may be divided into basic processing subunits in different schemes. Petition 870250092849, dated 10 / 10 / 2025, page 19 / 119 12 / 77

[0039] In some implementations, to provide parallel processing capability and resilience to errors in video encoding and decoding, an image can be divided into regions for processing, so that, for one region of the image, the encoding or decoding process does not depend on any information from any other regions of the image. In other words, each region of the image can be processed independently. By doing so, the codec can process different regions of an image in parallel, thus increasing encoding efficiency. Furthermore, when data from one region is corrupted during processing or lost in network transmission, the codec can correctly encode or decode other regions of the same image without depending on corrupted or lost data, thus providing error resilience. In some video encoding standards, an image can be divided into different types of regions. For example, H.265 / HEVC, H.266 / VVC and AVS provide two types of regions: “layers” and “blocks”. It should also be noted that different images in the 100-video sequence may have different partitioning schemes for dividing an image into regions.

[0040] For example, in FIG. 1, structure 110 is divided into three regions 114, 116, and 118, whose boundaries are shown as solid lines within structure 110. Region 114 includes four basic processing units. Each of regions 116 and 118 includes six basic processing units. It should be noted that the basic processing units, basic processing subunits, and regions of structure 110 in FIG. 1 are only examples, and the present invention does not limit their embodiments.

[0041] FIG. 2A illustrates a schematic diagram of an exemplary coding process 200A, consistent with the embodiments of the description. For example, the coding process 200A can be Petition 870250092849, dated 10 / 10 / 2025, p. 20 / 119 13 / 77 performed by an encoder. As shown in FIG. 2A, the encoder can encode a sequence of 202 videos into a 228 video bitstream according to process 200A. Similar to the sequence of 100 videos in FIG. 1, the sequence of 202 videos can include a set of images (referred to as “original images”) arranged in a temporal order. Similar to structure 110 in FIG. 1, each original image of the sequence of 202 videos can be divided by the encoder into basic processing units, basic processing subunits, or regions for processing. In some embodiments, the encoder can perform process 200A at the level of basic processing units for each individual image of the sequence of 202 videos. For example, the encoder can perform process 200A iteratively, where the encoder can encode one basic processing unit in one iteration of process 200A.In some modes, the encoder can execute process 200A in parallel for regions (e.g., regions 114-118) of each original image in the 202 video sequence.

[0042] In FIG. 2A, the encoder can feed a basic processing unit (referred to as an “original BPU”) of an original video sequence image 202 to prediction stage 204 to generate prediction data 206 and predicted BPU 208. The encoder can subtract the predicted BPU 208 from the original BPU to obtain the residual BPU 210. The encoder can feed the residual BPU 210 to the transform stage 212 and quantization stage 214 to generate quantized transform coefficients 216. The encoder can feed prediction data 206 and quantized transform coefficients 216 to the binary encoding stage 226 to generate the video bitstream 228. Components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 can be referred to as an “advance path”. During process 200A, after quantization stage 214, the encoder can feed Petition 870250092849, dated 10 / 10 / 2025, page 21 / 119 14 / 77 the quantized transform coefficients 216 for the inverse quantization stage 218 and inverse transform stage 220 to generate the reconstructed residual BPU 222. The encoder can add the reconstructed residual BPU 222 to the predicted BPU 208 to generate the prediction reference 224, which is used in the prediction stage 204 for the next process iteration 200A. The components 218, 220, 222, and 224 of process 200A can be referred to as a “reconstruction path”. The reconstruction path can be used to ensure that the encoder and decoder use the same reference data for prediction.

[0043] The encoder can execute process 200A iteratively to encode each original BPU of the original image (in the forward path) and generate the predicted reference 224 to encode the next original BPU of the original image (in the reconstruction path). After encoding the original BPUs of the original image, the encoder can proceed to encode the next image in the video sequence 202.

[0044] Referring to process 200A, the encoder may receive the video sequence 202 generated by a video capture device (e.g., a camera). The term “receive” as used in this document may refer to receiving, inserting, acquiring, retrieving, obtaining, reading, accessing, or any action in any way to insert data.

[0045] In prediction stage 204, in a current iteration, the encoder may receive an original BPU and prediction reference 224, and execute a prediction operation to generate prediction data 206 and predicted BPU 208. Prediction reference 224 may be generated from the reconstruction path of the previous iteration of process 200A. The purpose of prediction stage 204 is to reduce information redundancy by extracting prediction data 206 which can be used to reconstruct the original BPU as predicted BPU 208 from prediction data 206 and prediction reference 224. Petition 870250092849, dated 10 / 10 / 2025, page 22 / 119 15 / 77

[0046] Ideally, the predicted BPU 208 can be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 208 is generally slightly different from the original BPU. To account for these differences, after generating the predicted BPU 208, the encoder can subtract it from the original BPU to generate the residual BPU 210. For example, the encoder can subtract values ​​(e.g., grayscale values ​​or RGB values) from pixels of predicted BPU 208 from corresponding pixel values ​​of the original BPU. Each pixel of residual BPU 210 may have a residual value as a result of such subtraction between the corresponding pixels of the original BPU and the predicted BPU 208. Compared to the original BPU, the prediction data 206 and residual BPU 210 may have fewer bits, but they can be used to reconstruct the original BPU without significant quality deterioration. Thus, the original BPU is compressed.

[0047] To further compress the residual BPU 210, in the transform stage 212, the encoder can reduce the spatial redundancy of the residual BPU 210 by decomposing it into a set of two-dimensional “base patterns,” each base pattern being associated with a “transform coefficient.” The base patterns can be the same size (e.g., the size of the residual BPU 210). Each base pattern can represent a frequency variation component (e.g., brightness variation frequency) of the residual BPU 210. None of the base patterns can be reproduced from any combinations (e.g., linear combinations) of any other base patterns. In other words, the decomposition can decompose variations of the residual BPU 210 into a frequency domain.This decomposition is analogous to a discrete Fourier transform of a function, where the basis patterns are analogous to the basis functions (e.g., trigonometric functions) of the discrete Fourier transform. Petition 870250092849, dated 10 / 10 / 2025, page 23 / 119 16 / 77 and the transform coefficients are analogous to the coefficients associated with the basis functions.

[0048] Different transform algorithms can use different basis patterns. Several transform algorithms can be used in the 212 transform stage, such as, for example, a discrete cosine transform, a discrete sine transform, or similar ones. The transform in the 212 transform stage is reversible. That is, the encoder can restore the residual BPU 210 by an inverse transform operation (referred to as an “inverse transform”). For example, to restore a pixel from residual BPU 210, the inverse transform can multiply the corresponding pixel values ​​of the basis patterns by their respective associated coefficients and add the products to produce a weighted sum. For a video encoding pattern, both the encoder and the decoder can use the same transform algorithm (thus, the same basis patterns).Thus, the encoder can only record the transform coefficients, by which the decoder can reconstruct the residual BPU 210 without receiving the base patterns from the encoder. Compared to the residual BPU 210, the transform coefficients may have few bits, but they can be used to reconstruct the residual BPU 210 without significant quality deterioration. Thus, the residual BPU 210 is additionally compressed.

[0049] The encoder can further compress the transform coefficients at quantization stage 214. In the transform process, different basis patterns can represent different frequency variations (e.g., brightness variation frequencies). Since human eyes are generally better at recognizing low-frequency variation, the encoder can disregard high-frequency variation information without causing significant quality deterioration in the decoding. For example, at the stage of Petition 870250092849, dated 10 / 10 / 2025, p. 24 / 119 17 / 77 quantization 214, the encoder can generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (referred to as a “quantization scaling factor”) and rounding the quotient to its nearest integer. After such an operation, some transform coefficients of the high-frequency base patterns can be converted to zero, and the transform coefficients of the low-frequency base patterns can be converted to smaller integers. The encoder can disregard the zero-valued quantized transform coefficients 216, whereby the transform coefficients are further compressed. The quantization process is also reversible, in that the quantized transform coefficients 216 can be reconstructed into the transform coefficients in an inverse quantization operation (referred to as “inverse quantization”).

[0050] Since the encoder disregards the remainder of such divisions in the rounding operation, the quantization stage 214 may contain data loss. Typically, the quantization stage 214 may contribute to the greatest loss of information in process 200A. The greater the loss of information, the fewer bits of the quantized transform coefficients 216 are needed. To obtain different levels of loss of information, the encoder may use different values ​​of the quantization parameter or any other parameter of the quantization process.

[0051] In the binary coding stage 226, the encoder can encode the prediction data 206 and quantized transform coefficients 216 using a binary coding technique, such as, for example, entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other compression algorithm. Petition 870250092849, dated 10 / 10 / 2025, page 25 / 119 18 / 77 are either lossless or lossy. In some embodiments, in addition to the prediction data 206 and quantized transform coefficients 216, the encoder may encode other information in binary encoding stage 226, such as, for example, a prediction mode used in prediction stage 204, prediction operation parameters, a transform type in transform stage 212, quantization process parameters (e.g., quantization parameters), an encoder control parameter (e.g., a bit rate control parameter), or the like. The encoder may use the output data from binary encoding stage 226 to generate the video bitstream 228. In some embodiments, the video bitstream 228 may be further packaged for network transmission.

[0052] Referring to the reconstruction path of process 200A, at inverse quantization stage 218, the encoder can perform inverse quantization on the quantized transform coefficients 216 to generate reconstructed transform coefficients. At inverse transform stage 220, the encoder can generate reconstructed residual BPU 222 based on the reconstructed transform coefficients. The encoder can add reconstructed residual BPU 222 to the predicted BPU 208 to generate prediction reference 224, which is used in the next iteration of process 200A.

[0053] It should be noted that other variations of process 200A can be used to encode the video sequence 202. In some embodiments, the stages of process 200A can be performed by the encoder in different orders. In some embodiments, one or more stages of process 200A can be combined into a single stage. In some embodiments, a single stage of process 200A can be divided into multiple stages. For example, the stage of Petition 870250092849, dated 10 / 10 / 2025, page 26 / 119 19 / 77 transform 212 and quantization stage 214 can be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, process 200A may omit one or more stages in FIG. 2A.

[0054] FIG. 2B illustrates a schematic diagram of another exemplary coding process 200B, consistent with the modalities of the description. Process 200B can be modified from process 200A. For example, process 200B can be used by an encoder to conform to a hybrid video coding standard (e.g., H.26x series). Compared with process 200A, the forward path of process 200B additionally includes a mode decision stage 230 and divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B additionally includes a circuit filter stage 232 and a buffer stage 234.

[0055] Generally, prediction techniques can be categorized into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-image prediction or “intraprediction”) can use pixels from one or more surrounding BPUs already encoded in the same image to predict the current BPU. That is, the 224 prediction reference in spatial prediction can include the surrounding BPUs. Spatial prediction can reduce the inherent spatial redundancy of the image. Temporal prediction (e.g., inter-image prediction or “interprediction”) can use regions from one or more already encoded images to predict the current BPU. That is, the 224 prediction reference in temporal prediction can include the encoded images. Temporal prediction can reduce the inherent temporal redundancy of the images.

[0056] Referring to process 200B, in the forward path, the encoder performs the prediction operation in the spatial prediction stage. Petition 870250092849, dated 10 / 10 / 2025, p. 27 / 119 20 / 77 spatial 2042 and in the temporal prediction stage 2044. For example, in the spatial prediction stage 2042, the encoder can perform intraprediction. For an original BPU of an image being encoded, the prediction reference 224 may include one or more surrounding BPUs that were encoded (in the forward path) and reconstructed (in the reconstructed path) in the same image. The encoder can generate predicted BPU 208 by extrapolating the surrounding BPUs. The extrapolation technique may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, or similar. In some embodiments, the encoder can perform pixel-level extrapolation, such as extrapolating corresponding pixel values ​​for each pixel of the predicted BPU 208.The surrounding BPUs used for extrapolation can be located relative to the original BPU from various directions, such as in a vertical direction (e.g., at the top of the original BPU), a horizontal direction (e.g., to the left of the original BPU), a diagonal direction (e.g., below and to the left, below and to the right, above and to the left, or above and to the right of the original BPU), or any direction defined in the video encoding standard used. For intraprediction, the 206 prediction data may include, for example, locations (e.g., coordinates) of the surrounding BPUs used, sizes of the surrounding BPUs used, extrapolation parameters, a direction of the surrounding BPUs used relative to the original BPU, or similar.

[0057] For example, in the temporal prediction stage 2044, the encoder can perform interprediction. For an original BPU of a current image, the prediction reference 224 may include one or more images (referred to as “reference images”) that were encoded (in the forward path) and reconstructed (in the reconstructed path). In some modalities, a reference image may be encoded Petition 870250092849, dated 10 / 10 / 2025, p. 28 / 119 21 / 77 and the BPU reconstructed by the BPU. For example, the encoder can add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a reconstructed BPU. When all reconstructed BPUs from the same image are generated, the encoder can generate a reconstructed image as a reference image. The encoder can perform a “motion estimation” operation to search for a matching region within a scope (referred to as a “search window”) of the reference image. The location of the search window in the reference image can be determined based on the location of the original BPU in the current image. For example, the search window can be centered at a location with the same coordinates in the reference image as the original BPU in the current image and can be extended to a predetermined distance.When the encoder identifies (for example, using a pel-recursive algorithm, a block-matching algorithm, or similar) a region similar to the original BPU in the search window, the encoder can designate such a region as the matching region. The matching region may have different dimensions (for example, be smaller than, equal to, larger than, or in a different format) than the original BPU. Since the reference image and the current image are temporally separated on the timeline (for example, as shown in FIG. 1), the matching region can be considered to “move” to the location of the original BPU as time passes. The encoder can record the direction and distance of such movement as a “motion vector”. When multiple reference images are used (for example, an image 106 in FIG.1) The encoder can search for a matching region and determine its associated motion vector for each reference image. In some embodiments, the encoder can assign weights to the pixel values ​​of the matching regions of the respective corresponding reference images. Petition 870250092849, dated 10 / 10 / 2025, page 29 / 119 22 / 77

[0058] Motion estimation can be used to identify various types of motion, such as, for example, translations, rotations, zooms, or the like. For interprediction, the prediction data 206 may include, for example, locations (e.g., coordinates) of the matching region, motion vectors associated with the matching region, the number of reference images, weights associated with the reference images, or the like.

[0059] To generate the predicted BPU 208, the encoder can perform a “motion compensation” operation. Motion compensation can be used to reconstruct the predicted BPU 208 based on prediction data 206 (e.g., the motion vector) and the prediction reference 224. For example, the encoder can move the matching region of the reference image according to the motion vector, whereby the encoder can predict the original BPU of the current image. When multiple reference images are used (e.g., one image 106 in FIG. 1), the encoder can move the matching regions of the reference images according to the respective motion vectors and average pixel values ​​of the matching regions.In some modes, if the encoder has assigned weights to the pixel values ​​of the matching regions of the respective corresponding reference images, the encoder may add a weighted sum of the pixel values ​​of the moved matching regions.

[0060] In some modalities, interprediction can be unidirectional or bidirectional. Unidirectional interpredictions can use one or more reference images in the same temporal direction with respect to the current image. For example, image 104 in FIG. 1 is a unidirectionally interpredicted image, whose reference image (e.g., image 102) precedes image 104. Bidirectional interpredictions can use one or more reference images in both directions. Petition 870250092849, dated 10 / 10 / 2025, p. 30 / 119 23 / 77 temporal with respect to the current image. For example, image 106 in FIG. 1 is a bidirectional interpreted image, whose reference images (e.g., images 104 and 108) are in both temporal directions relative to image 104.

[0061] Still with reference to the path of progress of the process 200B, after the spatial prediction stage 2042 and temporal prediction stage 2044, in the mode decision stage 230, the encoder can select a prediction mode (e.g., one from intraprediction or interprediction) for the current iteration of process 200B. For example, the encoder can perform a rate distortion optimization technique, in which the encoder can select a prediction mode to minimize a value of a cost function depending on the bit rate of a candidate prediction mode and distortion of the reconstructed reference image under the candidate prediction mode. Depending on the selected prediction mode, the encoder can generate the predicted BPU 208 and predicted data 206.

[0062] In the reconstruction path of process 200B, if the intraprediction mode was selected in the advance path, after generating prediction reference 224 (e.g., the current BPU that was encoded and reconstructed in the current image), the encoder can directly feed prediction reference 224 to the spatial prediction stage 2042 for later use (e.g., for extrapolation of a next BPU from the current image). The encoder can feed prediction reference 224 to the circuit filter stage 232, where the encoder can apply a circuit filter to prediction reference 224 to reduce or eliminate distortion (e.g., blocking artifacts) introduced during the encoding of prediction reference 224. The encoder can apply various circuit filter techniques in the circuit filter stage 232, such as, for example, unblocking, offsetting. Petition 870250092849, dated 10 / 10 / 2025, page 31 / 119 24 / 77 adaptive sample coding, adaptive circuit filters, or the like. The circuit-filtered reference image may be stored in buffer 234 (or “decoded image buffer”) for later use (e.g., to be used as an interprediction reference image for a future video sequence image 202). The encoder may store one or more reference images in buffer 234 to be used in temporal prediction stage 2044. In some embodiments, the encoder may encode circuit filter parameters (e.g., a circuit filter intensity) in binary encoding stage 226, along with quantized transform coefficients 216, prediction data 206, and other information.

[0063] FIG. 3A illustrates a schematic diagram of an exemplary decoding process 300A, consistent with the embodiments described.Process 300A may be a decompression process corresponding to compression process 200A in FIG. 2A. In some embodiments, process 300A may be similar to the reconstruction path of process 200A. A decoder may decode the bitstream of videos 228 into video transmission 304 according to process 300A. Video transmission 304 may be very similar to the video sequence 202. However, due to information loss in the compression and decompression process (e.g., quantization stage 214 in FIGS. 2A-2B), video transmission 304 is generally not identical to the video sequence 202. Similar to processes 200A and 200B in FIGS. In versions 2A-2B, the decoder can execute process 300A at the basic processing unit (BPU) level for each image encoded in the 228-bit video stream.For example, the decoder can execute process 300A iteratively, where the decoder can decode a basic processing unit in one iteration of process 300A. In some embodiments, the decoder can execute process 300A. Petition 870250092849, dated 10 / 10 / 2025, page 32 / 119 25 / 77 in parallel for regions (e.g., regions 114-118) of each image encoded in the 228 bitstream.

[0064] In FIG. 3A, the decoder can feed a portion of the video bitstream 228 associated with a basic processing unit (referred to as a “coded BPU”) of an encoded image to the binary decoding stage 302. In the binary decoding stage 302, the decoder can decode the portion into prediction data 206 and quantized transform coefficients 216. The decoder can feed the quantized transform coefficients 216 to the inverse quantization stage 218 and inverse transform stage 220 to generate the reconstructed residual BPU 222. The decoder can feed prediction data 206 into the prediction stage 204 to generate the predicted BPU 208. The decoder can add the reconstructed residual BPU 222 to the predicted BPU 208 to generate the predicted reference 224. In some embodiments, the predicted reference 224 can be stored in a buffer (e.g., a decoded image buffer in computer memory.The decoder can feed predicted reference 224 to prediction stage 204 to perform a prediction operation in the next iteration of process 300A.

[0065] The decoder can execute process 300A iteratively to decode each encoded BPU of the encoded image and generate the predicted reference 224 to encode the next encoded BPU of the encoded image. After decoding all encoded BPUs of the encoded image, the decoder can send the image to the video transmission 304 to display and proceed to decode the next encoded image in the video bitstream 228.

[0066] In the binary decoding stage 302, the decoder can perform an inverse operation of the binary encoding technique. Petition 870250092849, dated 10 / 10 / 2025, page 33 / 119 26 / 77 used by the encoder (e.g., entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless data compression algorithm). In some embodiments, in addition to the prediction data 206 and quantized transform coefficients 216, the decoder may decode other information in binary decoding stage 302, such as, for example, a prediction mode, prediction operation parameters, a transform type, quantization process parameters (e.g., quantization parameters), an encoder control parameter (e.g., a bit rate control parameter), or the like. In some embodiments, if the video bitstream 228 is transmitted over a network in packets, the decoder may unpack the video bitstream 228 before feeding it into the binary decoding stage 302.

[0067] FIG. 3B illustrates a schematic diagram of another exemplary decoding process 300B, consistent with the modalities of the description. Process 300B can be modified from process 300A. For example, process 300B can be used by a decoder conforming to a hybrid video coding standard (e.g., H.26x series). Compared with process 300A, process 300B additionally divides the prediction stage 204 into spatial prediction stage 2042 and temporal prediction stage 2044, and additionally includes the circuit filter stage 232 and the buffer 234.

[0068] In process 300B, for a basic encoded processing unit (referred to as a “current BPU”) of an encoded image (referred to as a “current image”) being decoded, the prediction data 206 decoded from binary decoding stage 302 by the decoder may include several types Petition 870250092849, dated 10 / 10 / 2025, page 34 / 11927 / 77 of data, depending on which prediction mode the coder used to encode the current BPU. For example, if intraprediction was used by the coder to encode the current BPU, the 206 prediction data might include an indicator of the intraprediction prediction mode (e.g., an indicative value), parameters of the intraprediction operation, or similar. The parameters of the intraprediction operation might include, for example, locations (e.g., coordinates) of one or more surrounding BPUs used as a reference, sizes of the surrounding BPUs, extrapolation parameters, a direction of the surrounding BPUs with respect to the original BPU, or similar. In another example, if interprediction was used by the coder to encode the current BPU, the 206 prediction data might include an indicator of the interprediction prediction mode (e.g., an indicative value), parameters of the interprediction operation, or similar.The parameters of the interprediction operation may include, for example, the number of reference images associated with the current BPU, weights respectively associated with the reference images, locations (e.g., coordinates) of one or more matching regions in the respective reference images, one or more motion vectors respectively associated with the matching regions, or similar.

[0069] Based on the prediction mode indicator, the decoder can decide whether to perform a spatial prediction (e.g., intraprediction) at spatial prediction stage 2042 or a temporal prediction (e.g., interprediction) at temporal prediction stage 2044. The details of performing such a spatial or temporal prediction are described in FIG. 2B and will not be repeated herein. After performing such a spatial or temporal prediction, the decoder can generate the predicted BPU 208. The decoder can add the predicted BPU 208 and the reconstructed residual BPU 222 to generate prediction reference 224, as described in FIG. 3A. Petition 870250092849, dated 10 / 10 / 2025, page 35 / 119 28 / 77

[0070] In process 300B, the decoder can feed the predicted reference 224 to spatial prediction stage 2042 or temporal prediction stage 2044 to perform a prediction operation in the next iteration of process 300B. For example, if the current BPU is decoded using intraprediction in spatial prediction stage 2042, after generating prediction reference 224 (e.g., the decoded current BPU), the decoder can directly feed prediction reference 224 into spatial prediction stage 2042 for later use (e.g., for extrapolation of a next BPU from the current image).If the current BPU is decoded using interprediction in the time prediction stage 2044, after generating the prediction reference 224 (e.g., a reference image in which all BPUs have been decoded), the decoder can feed the prediction reference 224 into the circuit filter stage 232 to reduce or eliminate distortion (e.g., blocking artifacts). The decoder can apply a circuit filter to the prediction reference 224, in a manner described in FIG. 2B. The circuit-filtered reference image can be stored in buffer 234 (e.g., decoded image buffer in computer memory) for later use (e.g., to be used as an interprediction reference image for a future encoded video bitstream image 228). The decoder can store one or more reference images in buffer 234 for use in time prediction stage 2044.In some modes, the prediction data may additionally include circuit filter parameters (e.g., a circuit filter intensity). In some modes, the prediction data includes circuit filter parameters when the prediction mode indicator of prediction data 206 indicates that interprediction was used to encode the current BPU. Petition 870250092849, dated 10 / 10 / 2025, page 36 / 119 29 / 77

[0071] FIG. 4 is a block diagram of an example device 400 for encoding or decoding video, consistent with the embodiments described. As shown in FIG. 4, the device 400 may include the processor 402. When the processor 402 executes the instructions described here, the device 400 may become a specialized machine for encoding or decoding videos. The processor 402 may be any type of circuit capable of handling or processing information.For example, the 402 processor may include any combination of any number of a central processing unit (or “CPU”), a graphics processing unit (or “GPU”), a neural processing unit (“NPU”), a microcontroller unit (“MCU”), an optical processor, a programmable logic controller, a microcontroller, a microprocessor, a digital signal processor, an intellectual property (IP) core, a Programmable Logic Array (PLA), a programmable array logic (PAL), a Generic Array Logic (GAL), a Programmable Complex Logic Device (CPLD), a Field Programmable Gate Array (FPGA), a System on a Chip (SoC), an Application-Specific Integrated Circuit (ASIC), or the like. In some embodiments, the 402 processor may also be an array of processors grouped together as a single logic component. For example, as shown in FIG.4. The 402 processor may include multiple processors, including the 402a processor, the 402b processor, and the 402n processor.

[0072] The 400 device may also include 404 memory configured to store data (e.g., a set of instructions, computer code, intermediate data, or the like). For example, as shown in FIG. 4, the stored data may include program instructions (e.g., program instructions to implement the stages in processes 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequences). Petition 870250092849, dated 10 / 10 / 2025, page 37 / 119 30 / 77 202, video bitstream 228, or video transmission 304). The 402 processor can access program instructions and data for processing (e.g., via bus 410), and execute program instructions to perform an operation or manipulation on the data for processing. The 404 memory may include a high-speed random-access storage device or a non-volatile storage device. In some embodiments, the 404 memory may include any combination of any number of a random-access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive, a flash drive, a secure digital card (SD), a memory stick, a compact flash card (CF), or the like. The 404 memory may also be a group of memories (not shown in FIG. 4) grouped together as a single logical component.

[0073] The 410 bus can be a communication device that transfers data between components within the 400 appliance, such as an internal bus (e.g., a CPU-memory bus), an external bus (e.g., a universal serial bus port, a peripheral component interconnect express port), or the like.

[0074] To facilitate explanation without causing ambiguity, the 402 processor and other data processing circuits are collectively referred to as a “data processing circuit” in this description. The data processing circuit may be implemented entirely as hardware, or as a combination of software, hardware, or firmware. Furthermore, the data processing circuit may be a single standalone module or may be combined wholly or partially into any other component of the 400 appliance.

[0075] The 400 device may additionally include the interface of Petition 870250092849, dated 10 / 10 / 2025, page 38 / 119 31 / 77 406 network to provide wired or wireless communication with a network (e.g., the Internet, an intranet, a local area network, a mobile communications network, or the like). In some embodiments, the 406 network interface may include any combination of any number of a network interface controller (NIC), a radio frequency (RF) module, a transponder, a transceiver, a modem, a router, a gateway, a wired network adapter, a wireless network adapter, a Bluetooth adapter, an infrared adapter, a near-field communication (“NFC”) adapter, a cellular network chip, or the like.

[0076] In some embodiments, the 400 apparatus may optionally include a peripheral interface 408 to provide a connection for one or more peripheral devices. As shown in FIG. 4, the peripheral device may include, but is not limited to, a cursor control device (e.g., a mouse, a touchpad, or a touch screen), a keyboard, a display (e.g., a cathode ray tube display, a liquid crystal display, or a light-emitting diode display), a video input device (e.g., a camera or an input interface coupled to a video file), or the like.

[0077] It should be noted that video codecs (for example, a codec running processes 200A, 200B, 300A, or 300B) can be implemented as any combination of any software or hardware modules in the 400 device. For example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more software modules of the 400 device, such as program instructions that can be loaded into memory 404. For another example, some or all stages of process 200A, 200B, 300A, or 300B can be implemented as one or more Petition 870250092849, dated 10 / 10 / 2025, page 39 / 119 32 / 77 hardware modules of the 400 appliance, such as a specialized data processing circuit (e.g., an FPGA, an ASIC, an NPU, or similar).

[0078] The bidirectional optical flow (BDOF) tool has been adopted in the VVC standard. BDOF, formerly referred to as bidirectional optical flow, was proposed to improve the accuracy of optical flow-based interprediction, and was first adopted in the JEM software. Compared to the JEM version, BDOF in VVC is a simpler version that requires much less computation, especially in terms of the number of multiplications and the size of the multiplier.

[0079] In VVC, BDOF is used to refine the bi-prediction signal of a CU at the 4x4 sub-block level. BDOF is applied to a CU if it satisfies all of the following conditions: - The CU is encoded using "true" bi-prediction mode, meaning that one of the two reference images precedes the current image in the display order, and the other follows the current image in the display order. The distances (i.e., POC difference) from the two reference images to the current image are equal. Both reference images are short-term reference images. - The CU is not encoded using affine mode or the SbTMVP merge mode. CU has more than 64 luma samples. Both the height and width of the CU are greater than or equal to 8 luma samples. - Biprediction with a body weight index (BCW) indicates equal weight. Weighted Prediction (WP) is not enabled for the current CU. Petition 870250092849, dated 10 / 10 / 2025, page 40 / 119 33 / 77 The combined interprediction and intraprediction (CIIP) mode is not used for the current CU.

[0080] BDOF is applied only to the luma component. As the name indicates, the BDOF mode is based on the concept of optical flow, which assumes that the motion of an object is smooth. For each 4x4 sub-block, a motion refinement (vx,vy) is calculated by minimizing the difference between the prediction samples L0 and L1. The motion refinement is then used to adjust the bi-predicted sample values ​​in the 4x4 sub-block. The following steps are applied in the BDOF process.

[0081] Firstly, the horizontal and vertical gradients, dl(k)di™ _........ „ ... -^7 (i,j) and (i,j), k = 0.1, of the two prediction signals are calculated by directly calculating the difference between two neighboring samples, that is, dl(k)dx dl(k)dy (i,j) = (U(kKi + 1,j) » shift!) - (I(k\i (1) (i,j) = ((I(k\i,j + 1) » shiftl) — (I(k\i,j — 1) » shiftl)') where / (k)(i,j) is the sample value at coordinate (i,]) of the prediction signal in list k, k = 0.1, and shiftl is calculated based on the bit depth luma, bitDepth, as shiftl = max( 6, bitDepth-6).

[0082] Then, the autocorrelation and cross-correlation of the gradients, S1, S2, S3, S5 and S6, are calculated as Si = Σ Abs(px(i,j)), (íJW S2 = Σ x(i,D Slgn(ipy(l,j)) (i,j)EnS3 = Σθ(^ 'sl9n(ipx(ljy) (ij)en (2) Petition 870250092849, dated 10 / 10 / 2025, p. 41 / 119 34 / 77 S5 = ΣAbs(^y(i,j)) (ί,ηεη S 6= Σ ((i'jy SignWyii'j)) (ί,])^Ω where / d!(1)\ x(i'j^ = —(i>j) +^—(i,j)» n \ OX Xx / (dl(1)dlw\ Ψy(i,D = {-^(i'j)+~^(i,j)I »na Θ(i,j) = (lW(i,j) » nb) - (l(0^(i,j) » nb) (3) where Ω is a 6x6 window around the 4x4 sub-block, and the values ​​of nae and nb are defined as equal to min(1, bitDepth-11) and min(4, bitDepth-8), respectively, where min (x, y) is a function to obtain the smallest value between x and y.

[0083] The motion refinement (vx, vy) is then derived using the cross-correlation and autocorrelation terms as follows: vx=S1> 0? clip3 (-th^o, thBm,-((S3 2«»-«°) » llog2 ^1)):0 vy= S5 > 0? clip3 - ((^^2^) « ns2 + vxS2^) / 2) » [log2 Sj)) : 0 where S2.rn = s2» n2, S2X= S&(2«· - 1), th'BI0= 2m“<5·™-7). H is the floor function, ens2= 12.

[0084] Based on motion refinement and gradients, the following fit is calculated for each sample in the 4x4 sub-block: (-th'BIo, thBIo, - ((S6 · 2«» «° Petition 870250092849, dated 10 / 10 / 2025, p. 42 / 119 35 / 77 ) (5) dl(1)(x,y) δίΟζχ,γ) ------------------------------------------------d xdx (dl(1)(x,y) dlw(x,y)\\ \ '

[0085] Finally, the CU BDOF samples are calculated by adjusting the bi-prediction samples as follows: predBD0F(x,y) = ( / (0)(x,y) + / (1)(x,y) + b(x,y) + ooffset) (6) >> shift

[0086] These values ​​are selected so that the multipliers in the BDOF process do not exceed 15 bits, and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits.

[0087] In order to derive the gradient values, some prediction samples I^ti, / ) in the list k (k = 0,1) outside the boundaries of the current CU need to be generated. As depicted in FIG. 5, the BDOF in the VVC uses an extended row / column around the CU boundaries. In order to control the computational complexity of generating the prediction samples outside the boundaries, the prediction samples in the extended area (white positions) are generated by obtaining the reference samples at the nearest integer positions (using the floor() operation on the coordinates) directly without interpolation, and the 8-touch motion compensation normal interpolation filter is used to generate prediction samples within the CU (gray positions). The extended sample values ​​are used only in the gradient calculation.For the remaining steps in the BDOF process, if any of the sample and gradient values ​​outside the CU boundaries are needed, they are filled in (i.e., repeated) from their nearest neighbors.

[0088] When the width or height of a UC is greater than 16 Petition 870250092849, dated 10 / 10 / 2025, p. 43 / 119 36 / 77 luma samples, it will be divided into sub-blocks with a width or height equal to 16 luma samples, and the sub-block boundaries are treated as the CU boundaries in the BDOF process. The maximum unit size for the BDOF process is limited to 16*16. For each sub-block, the BDOF process can be skipped. When the SAD between the initial prediction samples L0 and L1 is less than a threshold, the BDOF process is not applied to the sub-block. The threshold is defined as equal to (8*W*(H>>1), where W indicates the width of the sub-block, and H indicates the height of the sub-block. To avoid the additional complexity of calculating the SAD, the SAD between the initial prediction samples L0 and L1 calculated in the DMVR process is reused here.

[0089] If BCW is enabled for the current block, i.e., the BCW weight index indicates uneven weight, then bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, i.e., luma_weight_lx_flag is 1 for either of the two reference images, then BDOF is also disabled. When CU is encoded with symmetric MVD mode or CIIP mode, BDOF is also disabled.

[0090] In ECM, to further improve encoding performance, a sample-based BDOF is adopted. In sample-based BDOF, instead of deriving the (yx, vy) block-based motion refinement, it is performed sample-wise.

[0091] In addition to sample-level motion refinement calculation, for samples outside the CU boundaries, true sample values ​​are used instead of filled values. Thus, the decoder needs to retrieve and interpolate a larger sample area than the current CU for gradient calculation. Similarly, since the true sample value is used for samples outside the boundaries, there is no need to divide the CU into a 16*16 sub-block to fill the boundary. Petition 870250092849, dated 10 / 10 / 2025, page 44 / 119 37 / 77

[0092] For boundary verification, the coding block is divided into 8*8 sub-blocks. For each sub-block, the decision to apply or not apply BDOF is determined by checking the SAD between the two reference sub-blocks with respect to the boundary. If it is decided to apply BDOF to a sub-block, for each sample in the sub-block, a 5*5 moving window is used and the existing BDOF process is applied to each moving window to derive vx and vy. The derived motion refinement (vx, vy) is applied to adjust the bi-predicted sample value to the central sample of the window.

[0093] Since the 8-touch interpolation filter used in VVC is replaced by a 12-touch filter, BDOF also uses the 12-touch interpolation filter to generate the predicted samples.

[0094] The high-precision BDOF is described below. To improve the accuracy of the motion refinement calculation, the following equations are used to replace equations (4).

[0095] First, horizontal and vertical gradients, dl(k)dl^z. _........- - ... ---(i,j) and ---(i,]), k = 0,1, of the two prediction signals are calculated dx yy by directly calculating the difference between two neighboring samples, that is, dl(k)zz„ (i,]) = (G(k)(i + 1,]) » shift!) - (I(k\i - 1,]) » shift!)') dl(k)zz„ —(i,D = ((I(kKi,j + 1) » shiftlj - (I{kKi,i - 1) » shift1)) (7) where / (k)(ij) are the sample value at coordinate (i,]) of the prediction signal in list k, k = 0,1, and shiftl is calculated based on the luma bit depth.

[0096] Then, the autocorrelation and cross-correlation of the gradients, S1, S2, S3, S4, S5 and S6, are calculated as Petition 870250092849, dated 10 / 10 / 2025, p. 45 / 119 38 / 77 5i = Σ Wíj · ijjx(i,j) · xpx(i,J^, (ij)^n S2 = S4 Σ Wíj · ^x(i,j) · ^y(i,j) (i.Dea Σ Wíj · 0(i,j) •^x(i,j) (ij)en (8) Wíj •^y(i,j) · ^Py(i,j^ where 6=Σ Wíj · θ(ΐ,β · ipy(i,j) (ij)en / dI(1)dIW\ x(i,j) = 3—(i,j) +^—(i,j)» n \ OX ox / / dI(1)y(i,j= \~d^(i,j^+~d^~(i,j)»na(9) Θ(i,j) = (lW(i,j) » nb) - (l(0^(i,j^ » nb)) where wij is a position-dependent weight with the position (i, j) closest to the current sample, the value of wij being greater, and Ω is a 5x5 window around the current sample, and the values ​​of na and nb are defined as equal to min(1, bitDepth-11) and min(4, bitDepth-8), respectively, where min(x, y) is a function to obtain the smallest value between x and y. y.

[0097] In some implementations, regulation is applied by adding a compensation to 51 and S5 before the motion refinement derivation (vx, vy). The regulation can be expressed as: (10) Petition 870250092849, dated 10 / 10 / 2025, p. 46 / 119 39 / 77 where d is the compensation whose value could be dependent on the size of the sub-block.

[0098] The motion refinement (vx, vy) is then derived using the cross-correlation and autocorrelation terms as follows: Vx = 3^5 & S 2$6 S 2S4vy = S 1^6 51S5 3^4 2.S4 (11)

[0099] Based on motion refinement and gradients, the following adjustment is calculated for each sample. fí / b(x,y) = rnd II vxI dl(1)(x,y) â / (0)(x,y) ^Kx.y) d / ^fey^ + ---—) + 1) / 2) (12)

[0100] Finally, the CU BDOF samples are calculated by adjusting the bi-prediction samples as follows: predBD0F(x,y)= (lW(x,y) + Iw(x,y) + b(x,y) + ooffset) » shift (13)

[0101] The decoder-oriented motion vector refinement (DMVR) will be described below. VVC adopts a decoder-oriented motion vector refinement based on bilateral matching (BM) in the bi-prediction operation to increase the accuracy of the merge mode MVs. In DMVR, a refined MV is searched around the initial MVs in the L0 list of the reference image and in the L1 list of the reference image. The BM method calculates the distortion between the two candidate blocks in the L0 list and L1 list of the reference image. As illustrated in FIG. 6, the SAD between the red blocks is calculated based on each candidate MV around the initial MV. Petition 870250092849, dated 10 / 10 / 2025, page 47 / 119 40 / 77 The MV candidate with the lowest SAD value becomes the refined MV and is used to generate the bipredicted signal.

[0102] In VVC, DMVR application is restricted and applied only to CUs that are coded with the following modes and characteristics: - CU-level merging mode with bi-prediction MV One reference image is in the past, and another reference image is in the future relative to the current image. The distances (i.e., POC difference) from the two reference images to the current image are equal. Both reference images are short-term reference images. CU has more than 64 luma samples. Both the height and width of the CU are greater than or equal to 8 luma samples. - BCW weight index indicates equal weight WP is not enabled for the current block. CIIP mode is not enabled for the current block.

[0103] The refined MV derived by the DMVR process is used to generate the interprediction samples and is also used in temporal motion vector prediction for future image coding. While the original MV is used in the unlocking process and is also used in spatial motion vector prediction for future CU coding.

[0104] The additional features of DMVR are mentioned in the following subclauses.

[0105] The search scheme used in DMVR is described below. In DMVR, search points are circled in the initial MV, and MV compensation follows the MV difference mirroring rule. In other words, any points that are checked by DMVR, Petition 870250092849, dated 10 / 10 / 2025, page 48 / 119 41 / 77 indicated by the candidate MV pair (MV0, MV1) obey the following two equations: MV 0' = MV0 + MV_offset (14) MV 1' = MV1 - MV_offset (15) where MV_o / / set represents the refinement offset between the initial MV and the refined MV in one of the reference images. The refinement search range is two luma samples of integers from the initial MV. The search includes the integer sample offset search stage and the fractional sample refinement stage.

[0106] A full 25-point search is applied for whole-sample compensation search. The SAD of the initial MV pair is calculated first. If the SAD of the initial MV pair is less than a threshold, the whole-sample DMVR stage is terminated. Otherwise, the SADs of the remaining 24 points are calculated and checked in raster scan order. The point with the lowest SAD value is selected as the output of the whole-sample compensation search stage. To reduce the uncertainty penalty of DMVR refinement, it is proposed to favor the original MV during the DMVR process. The SAD between the reference blocks mentioned by the initial MV candidates is decreased by 1 / 4 of the SAD value.

[0107] The whole sample search is followed by fractional sample refinement. To reduce the complexity of the calculation, the fractional sample refinement is derived using a parametric error surface equation, instead of additional search with SAD comparison. The fractional sample refinement is conditionally invoked based on the output of the whole sample search stage. When the whole sample search stage ends with the center having the lowest SAD value in the first iteration search or the second iteration search, the fractional sample refinement is additionally applied. Petition 870250092849, dated 10 / 10 / 2025, page 49 / 119 42 / 77

[0108] In the subpixel offset estimation based on parametric error surface, the cost of the center position and the costs of four surrounding positions from the center are used to fit the 2-D parabolic error surface equation as follows: E(x,y) = A(x - xmin)2+ B(y-ymin)2+ C (16) where (xmin,ymin) corresponds to the fractional position with the lowest cost and C corresponds to the minimum cost value. When solving the equations above using the cost value of the five search points, (xmin,ymin) is calculated as: xmin = (£(-1,0) - £(1,0)) / (2(£(-1,0) + £(1,0) - 2E(0,0^)) (17) ymin = (E(0, -1) - £(0,1)) / (2((£(0, -1) + £(0,1) - 2£(0,0))) (18)

[0109] The xmin and ymin values ​​are automatically constrained between -8 and 8, provided all cost values ​​are positive and the smallest value is £(0,0). This corresponds to half-pixel compensation with MV precision of 1 / 16-pel in VVC. The calculated fractional values ​​(xmin, ymin) are added to the integer distance refinement MV to obtain the subpixel precision refinement delta MV.

[0110] The following will describe bilinear interpolation and sample filling. In VVC, the resolution of the MVs is 1 / 16 luma samples. Samples in the fractional position are interpolated using an 8-touch interpolation filter. In DMVR, the search points are encircling the initial fractional MV with whole-sample compensation, therefore, samples from those fractional positions need to be interpolated for the DMVR search process. To reduce calculation complexity, the bilinear interpolation filter is used to generate the fractional samples for the search process in DMVR. Another important effect of using the bilinear filter is that, with a search interval of 2 samples, DMVR no longer accesses reference samples compared to the normal motion compensation process. After the Petition 870250092849, dated 10 / 10 / 2025, page 50 / 119 43 / 77 Once the refined MV is obtained using the DMVR search process, the normal 8-touch interpolation filter is applied to generate the final prediction. To avoid accessing reference samples again in the normal MC process, samples that are not needed for the interpolation process based on the original MV, but are needed for the interpolation process based on the refined MV, will be populated from those available samples.

[0111] Next, the maximum DMVR processing unit will be described. When the width or height of a CU is greater than 16 luma samples, it will be further divided into sub-blocks with a width or height equal to 16 luma samples. The maximum unit size for the DMVR search process is limited to 16^16.

[0112] The decoder-focused multi-step motion vector refinement (MP-DMVR) will be described below. In ECM, to further improve coding efficiency, a decoder-focused multi-step motion vector refinement is applied. In the first step, bilateral matching (BM) is applied to the coding block. In the second step, BM is applied to each 16x16 sub-block within the coding block. In the third step, MV in each 8x8 sub-block is refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for spatial and temporal motion vector prediction.

[0113] In the first step, block-based bilateral matching MV refinement is performed. In the first step, a refined MV is derived by applying BM to a coding block. Similar to decoder-side motion vector refinement (DMVR), in the bi-prediction operation, a refined MV is searched around two initial MVs (MV0 and MV1) and the L0 and L1 lists of reference images. The refined MVs (MV0_pass1 and MV1_pass1) are derived Petition 870250092849, dated 10 / 10 / 2025, page 51 / 119 44 / 77 vados around the initial MVs based on the minimum bilateral matching cost between the two reference blocks in L0 and L1.

[0114] BM performs local search to derive intDeltaMV of integer sample precision. The local search applies a 3^3 square search pattern to traverse the search range [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8 or other values. For example, as in FIG. 7, point 0 is the position to which the initial MV refers. Thus, points 1 to 8 around the initial position are searched first, and the cost of each position is calculated. If point 7 is the minimum cost, then point 7 is defined as the center of the search, and points 9, 10, and 11 are searched. If the cost of point 10 is less than point 7, the search center shifts to point 10, and points 12, 13, and 14 are searched. If point 12 has the minimum cost among points 6 to 14, point 12 is defined as the new search center.If points 10, 11, 13, 15 to 19 around point 12 are all greater than point 10, then point 12 is the best position and the search process ends.

[0115] The bilateral matching cost is calculated as: bilCost = mvDistanceCost + sadCost, where sadCost is the SAD between predictor l0 and predictor l1 at each search point and mvDistanceCost is based on intDeltaMV (i.e., the distance between the search point and the initial position). When the cbW * cbH block size is greater than 64, the MRSAD cost function is applied to remove the DC distortion effect between the reference blocks. When the bilCost at the center point of the 3x3 search pattern has the minimum cost, the local search intDeltaMV is terminated. Otherwise, the current minimum cost search point becomes the new center point of the 3x3 search pattern and continues the search for the minimum cost until the end of the search range is reached. Petition 870250092849, dated 10 / 10 / 2025, page 52 / 119 45 / 77

[0116] The existing fractional sample refinement is additionally applied to derive the final deltaMV. The refined MVs after the first step are then derived as: MV 0pass1= MV0 + deltaMV MVípass1= MVí - deltaMV (19)

[0117] In the second step, the MV refinement of sub-block-based bilateral matching is performed. In the second step, a refined MV is derived by applying BM to a 16^16 mesh sub-block. For each sub-block, a refined MV is searched around two MVs (MV0_pass1 and MV1_pass1), obtained in the first step, in the L0 and L1 lists of the reference image. The refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the cost of the minimum bilateral match between the two reference sub-blocks in L0 and L1.

[0118] For each sub-block, BM performs a full search to derive the integer sample precision intDeltaMV(sbIdx2). The full search has a search range [-sHor, sHor] in the horizontal direction and [sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8 or other values.

[0119] The bilateral matching cost is calculated by applying the cost factor to the SATD cost between two reference sub-blocks, as: bilCost = satdCost * costFactor. The search area (2*sHor + 1) * (2*sVer + 1) is divided into up to 5 diamond-shaped search regions shown in FIG. 8. Each search region is assigned a costFactor, which is determined by the intDeltaMV(sbIdx2) distance between each search point and the initial MV, and each diamond region is processed in order starting from the center of the search area. In each region, the search points are processed in raster scan order starting from the upper left corner and going to the lower right corner of the Petition 870250092849, dated 10 / 10 / 2025, page 53 / 119 46 / 77 region. When the minimum bilCost within the current search region is less than a limit equal to sbW * sbH, the full int-pel search is terminated; otherwise, the full int-pel search continues to the next search region until all search points have been examined. Additionally, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than a limit equal to the block area, the search process is terminated.

[0120] The existing VVC DMVR fractional sample refinement is additionally applied to derive the final deltaMV(sbIdx2). The MVs refined in the second step are then derived as: • Mr0pass2(sb / dx2) = MV0pass1+ deltaMV(sbIdx2) (20) • Mripass2(sb / dx2) = MV1pass1- deltaMV(sbIdx2) where sbIdx2 is the sub-block index for the second stage of multi-stage DMVR

[0121] In the third step, sub-block-based bidirectional optical flow MV refinement is performed. In the third step, a refined MV is derived by applying BDOF to an 8*8 mesh sub-block. For each 8*8 sub-block, the BDOF refinement process is applied to derive motion refinement (vx, vy) without clipping from the refined MV of the source sub-block from the second step. The method described above can be used. For example, equations (1) to (4) or equations (7) to (10) are invoked to derive (vx,vy), indicated as bioMV. After motion refinement derivation, bioMV is rounded to a sample accuracy of 1 / 16 and clipped between -32 and 32.

[0122] The refined MVs (MV0_pass3(sbIdx3) and MV1_pass3 (sbIdx3)) in the third step are derived as: • MV0pass3(sb / dx3) = MV0pass2(sbIdx2) + bioMV (21) • MV 1pass3(sb / dx3) = MV1pass2(sbIdx2') — bioMV where sbIdx2 is the sub-block index for the second stage of the DMVR Petition 870250092849, dated 10 / 10 / 2025, page 54 / 119 47 / 77 multi-stage and sbIdx3 is the sub-block index for the third stage of the multi-stage DMVR.

[0123] The decoder-facing motion vector refinement will be described below. In ECM, the decoder-facing motion vector refinement method is an extension of the multi-step DMVR that consists of two new merging modes to refine the MV in only one direction, either L0 or L1, of the bi-prediction for merging candidates that meet the DMVR conditions. The multi-step DMVR process is applied to the selected merging candidate to refine the motion vectors; however, MVD0 or MVD1 is set to zero in the 1st-step DMVR (i.e., PU level). Thus, a new list of merging candidates is built for the decoder-facing adaptive motion vector refinement. And the new merging mode for the new list of merging candidates is called BM merging in ECM.

[0124] Merge candidates for merge mode BMs are derived from spatially neighboring coded blocks, TMVPs, non-adjacent blocks, history-based motion vector predictors (HMVPs), and paired candidates, similar to the regular merge mode. The difference is that only those that meet the DMVR conditions are added to the candidate list. The same merge candidate list is used by both new merge modes. If the BM candidate list contains the inherited BCW weights, the DMVR process is unchanged, except that the distortion calculation is performed using MRSAD or MRSATD if the weights are not equal, and bi-prediction is weighted with the BCW weights. The merge index is coded as in the regular merge mode.

[0125] In the current multi-step DMVR, the first and second steps are based on bilateral matching. Bilateral matching Petition 870250092849, dated 10 / 10 / 2025, page 55 / 119 CU-level 48 / 77 matching is performed in the first stage, followed by sub-block-level bilateral matching in the second stage. CU-level bilateral matching refines the motion vector at a coarse level, while sub-block-level bilateral matching refines it at a finer level. However, for the third stage of multi-stage DMVR, which is based on optical flow, the refinement mesh is fixed, and there is only one stage for it. Smaller sub-block-level refinement provides finer refinement, while larger sub-block-level refinement provides greater resilience to errors and thus results in more stable optical flow calculations. Therefore, a single optical flow-based refinement stage on the fixed mesh cannot achieve accurate and stable results. Furthermore, depending on the different video content, the optimal sub-block size for motion vector refinement is variable.For example, smooth content might prefer refinement at the large sub-block level, while sophisticated content might require small, sub-block-level refinement. Therefore, motion vector refinement on a fixed mesh may not be suitable for all video content.

[0126] The present invention provides methods for solving the above problems. In particular, it is proposed to contain multiple steps for optical flow-based motion vector refinement. First, divide the CU into large sub-blocks and refinement is performed at the large sub-block level. After that, the large sub-blocks are further divided into smaller sub-blocks and based on the results of the first refinement step, a second refinement step is performed at the smaller sub-block level. In this method, the refinement of the first step provides a base refinement for the large sub-block and then, for each smaller sub-block within the large sub-block, a more precise motion vector refinement can be achieved. Petition 870250092849, dated 10 / 10 / 2025, page 56 / 119 49 / 77 obtained through the second refinement stage. For smooth video content, the refinement from the first stage could bring sufficient improvement to the motion vector, and for sophisticated video content, the refinement from the second stage will provide more benefits. Thus, both smooth and sophisticated video content could benefit from the proposed method.

[0127] However, refinement at the smaller sub-block level presents much greater complexity. To reduce complexity, after the first refinement step at the large sub-block level, no further sub-block splitting will be performed. That is, the second refinement step is performed on the sub-block with the same size as the first refinement step. In this case, after splitting the CU into sub-blocks, two refinement steps are performed on each sub-block. The first refinement step derives from a first motion refinement, and based on the refined motion, motion compensation is performed on each sub-block and optical flow-based refinement is again performed to find a second motion refinement. The final refined motion is obtained by adding the first and second motion refinements.

[0128] FIG. 9 is a flowchart of an exemplary method 900 for performing motion vector refinement and motion compensation, according to some embodiments of the present invention. Method 900 can be performed by an encoder (for example, by process 200A of FIG. 2A or 200B of FIG. 2B) or a decoder (for example, by process 300A of FIG. 3A or 300B of FIG. 3B) or performed by one or more software or hardware components of an apparatus (for example, apparatus 400 of FIG. 4). For example, one or more processors (for example, processor 402 of FIG. 4) can perform method 900. In some embodiments, the mé Petition 870250092849, dated 10 / 10 / 2025, page 57 / 119 50 / 77 all 900 can be implemented by a computer program product, embedded in a computer-readable medium, including computer-executable instructions, such as program code, executed by computers (e.g., device 400 of FIG. 4). As shown in FIG. 9, method 900 includes the following steps 910-950.

[0129] In step 910, the processor performs block-level bilateral matching. Block-level bilateral matching is performed on a coding block, and may be the first step in the decoder-facing multi-step motion vector refinement (MP-DMVR) process.

[0130] In step 920, the processor divides the coding block into NχN sub-blocks and performs two-sided matching at the sub-block level. This step can be the second step of the MP-DMVR described above. For example, the coding block can have a size of 64χ64, and the processor can divide the coding block into a mesh of 16χ16 sub-blocks. The processor can then apply two-sided matching to the plurality of 16χ16 sub-blocks.

[0131] In step 930, the processor divides the coding block into LχL sub-blocks and performs optical flow-based motion vector refinement on the LχL sub-blocks. This step can be the third step of the MP-DMVR described above—the sub-block level bidirectional optical flow motion vector refinement (BDOF) process. For example, the coding block might have a size of 64χ64, and the processor might divide the coding block into a mesh of 16χ16 or 8χ8 sub-blocks. The processor can then apply the BDOF motion vector refinement process to the plurality of 16χ16 or 8χ8 sub-blocks.

[0132] In step 940, the processor divides the coding block into MχM sub-blocks and performs motion vector refinement. Petition 870250092849, dated 10 / 10 / 2025, page 58 / 119 51 / 77 based on optical flow in M*M sub-blocks. Consistent with the disclosed modalities, the processor can repeat the third MP-DMVR step described above multiple times in different sized sub-blocks. Specifically, in step 940, each of the M*M sub-blocks is created smaller than each of the L*L sub-blocks, i.e., L>M. For example, the L*L sub-blocks can be 8*8 sub-blocks, while the M*M sub-blocks can be 4*4 or 2*2 sub-blocks.

[0133] In step 950, the processor performs sample-based bidirectional optical flow motion compensation.

[0134] Although method 900 executes the BDOF motion vector refinement process twice (i.e., executes the third MP-DMVR step twice), it is contemplated that the disclosed embodiments may execute this process in any number of steps, each step in different scaled sub-blocks. For example, method 900 may be modified to include an additional step 942 (not shown in FIG. 9) after step 940 and before step 950. In step 942, the processor may divide the coding block into a plurality of 2*2 sub-blocks, and execute a step of the BDOF motion vector refinement process in a plurality of 2*2 sub-blocks.

[0135] FIG. 10 is a flowchart of an exemplary method. 1000 to perform motion vector refinement and motion compensation, according to some embodiments of the present invention. Method 1000 can be performed by an encoder (for example, by process 200A of FIG. 2A or 200B of FIG. 2B) or a decoder (for example, by process 300A of FIG. 3A or 300B of FIG. 3B) or performed by one or more software or hardware components of an apparatus (for example, apparatus 400 of FIG. 4). For example, one or more processors (for example, processor 402 of Petition 870250092849, dated 10 / 10 / 2025, page 59 / 119 52 / 77 FIG. 4) can execute method 1000. In some embodiments, method 1000 can be implemented by a computer program product, embedded in a computer-readable medium, including computer-executable instructions, such as program code, executed by computers (e.g., device 400 of FIG. 4).

[0136] As shown in FIG. 10, method 1000 includes steps 1010-1050. Steps 1010, 1020, and 1050 are identical to steps 910, 920, and 950, respectively. Method 1000 differs from method 900 by performing a first step of the BDOF motion vector refinement process on the MχM sub-blocks (step 1030), and then performs a second step of the BDOF motion vector refinement process on the LχL sub-blocks (step 1040), where L>M.

[0137] Additionally, in some embodiments, the size of the adaptive sub-block can be used for optical flow-based motion vector refinement. For example, FIG. 11 is a flowchart of an exemplary method 1100 for performing motion vector refinement and motion compensation according to some embodiments of the present invention. Method 1100 can be performed by an encoder (e.g., by process 200A of FIG. 2A or 200B of FIG. 2B) or a decoder (e.g., by process 300A of FIG. 3A or 300B of FIG. 3B) or performed by one or more software or hardware components of an apparatus (e.g., apparatus 400 of FIG. 4). For example, one or more processors (e.g., processor 402 of FIG. 4) can perform method 1100.In some embodiments, method 1100 can be implemented by a computer program product, embedded in a computer-readable medium, including computer-executable instructions, such as program code, executed by computers (e.g., device 400 of FIG. 4). As shown in FIG. 11, method 1100 includes the following steps 1110-1160. Petition 870250092849, dated 10 / 10 / 2025, pp. 60 / 119 53 / 77

[0138] Specifically, step 1110 is the same as steps 910 and 1010, and step 1120 is the same as steps 920 and 1020, the details of which will not be repeated in this document.

[0139] In step 1130, the processor checks the subblock size (resulting from step 1120) against a condition. If the condition is met, the processor performs optical flow-based motion vector refinement at a large subblock level, for example, LχL subblocks (step 1140); and if the condition is not met, the processor performs optical flow-based motion vector refinement at a small subblock level, for example, MχM subblocks (step 1150). In the disclosed embodiments, the condition may be based on the coding block area, quantization parameters, or video resolution.For example, the condition could be one of the following: the coding block is coded using a bi-prediction mode; the coding block has a size that exceeds a predetermined limit; the coding block has a size smaller than a predetermined limit; weighted prediction is disabled for the coding block; combined interprediction and intraprediction (CIIP) is disabled for the coding block; local luma compensation is enabled for the coding block; sub-block motion compensation is applied to the coding block; symmetric motion vector difference is not used; or the coding block is not coded as a merge mode with motion vector difference.

[0140] Additionally, in step 1160, the processor performs sample-based bidirectional optical flow motion compensation. Step 1160 is the same as steps 950 and 1050.

[0141] The algorithm for implementing the optical flow-based motion refinement method is described in detail below. Specifically, in some embodiments, after the second stage of Petition 870250092849, dated 10 / 10 / 2025, pp. 61 / 119 54 / 77 In multi-step DMVR, the refined motion vectors MV0pass2 and MV1pass2 are obtained for each 16*16 sub-block, where MV0pass2 is a motion vector from list 0 of the reference image and MV1pass2 is a motion vector from list 1 of the reference image. Then, based on the refined motion vectors MV0pass2 and MV1pass2, interprediction is performed and Pred0_sb16ext and Pred1_sb16ext are obtained, where Pred0_sb16ext is the predicted block from list 0 of the reference image using MV0pass2 and Pred1_sb16ext is the predicted block from list 1 of the reference image using MV1pass2. In interprediction, the interpolation filter could be applied to generate the predicted pixel values ​​at the sub-block positions. 8-touch, 12-touch, or other interpolation filters can be used. Since gradients are needed in the optical flow equations, the predicted block is then extended. In this example, as shown in FIG.In Figure 12A, the 3-pixel rows or 3-pixel columns are extended on the four edges of the 16*16 subblock. In some other embodiments, in order to reduce bandwidth and interpolation calculation, the extended pixels (i.e., the pixels outside the subblock boundaries) may be filled from the boundary pixels within the 16*16 subblock, instead of actually performing interpolation on them, as shown in FIG. 5. In some embodiments, to reduce calculation, the refined motion vectors of two surrounding 16*16 subblocks are checked. If two surrounding 16*16 sub-blocks have the same refined motion vectors (i.e., MV0pass2(sb / d%2) is equal to MV0pass2(sb / d%1) and MV 1pass2(sb!dx2) is equal to MV1pass2(sb / dx2)), the two surrounding 16*16 sub-blocks can be merged into a 16*32 or 32*16 sub-block. Then, the motion vector of the merged sub-block will be continuously compared to the refined motion vectors of the new surrounding 16*16 sub-block.If the motion vectors are still... Petition 870250092849, dated 10 / 10 / 2025, page 62 / 119 55 / 77 are the same, so these two sub-blocks are additionally merged. The merging process is repeated to find the largest possible sub-block. Then, interprediction is performed on the merged sub-block. As a first step of optical flow-based motion refinement is performed at the 8*8 sub-block level, in some other modalities, interprediction could also be performed for each 8*8 sub-block within the 16*16 sub-block separately. This will not affect the results.

[0142] After the two predicted blocks are obtained. Then, for each 8*8 sub-block in the 16*16 sub-block, the optical flow-based motion vector refinement method can be applied. For example, equations (1) to (4) or equations (7) to (11) are used to derive the motion refinement fíVfiratPass= (vx_firstPass,vy_firstPass') for the first step. Similar to BDOF, the current SAD or SATD between Pred0_sb8 and Pred1_sb8 can be calculated and compared to a threshold, where Pred0_sb8 is the predicted sub-block of the current 8*8 sub-block from list 0 of the reference image, which is a sub-block of Pred0_sb16ext, and Pred1_sb8 is the predicted sub-block of the current 8*8 sub-block from list 1 of the reference image, which is a sub-block of Pred1_sb16ext. If the SAD is less than the threshold, the refinement process is skipped for this 8*8 sub-block.The limit, for example, can be defined as k*sbWidth*sbHeight, where sbWidth and sbHeight are the width and height of the predicted sub-block for the SAD or SATD calculation, and k is a positive factor. As shown in FIG. 12B, for an 8*8 sub-block, the pixels in the extended 14*14 area are needed to derive the motion refinement MVfírstPassatravés from equations (1) to (4) or equations (7) to (11). After deriving the motion refinement from the first step of the optical flow-based refinement, the motion vector of each 8*8 sub-block can be derived as: Petition 870250092849, dated 10 / 10 / 2025, page 63 / 119 56 / 77 • M70oFpaSS1(sb / dx3) = MV0pass2(sbIdx2) + MVfirstPass(sbIdx3) • M71OFpaSSi(sb / dx3) = MV1paSs2(sb / dx2) - MVfirStpaSS(sbIdx3) where M70OFpass1 and MV1OFpass1 are the motion vector refined by the first step of optical flow-based refinement for reference image list 0 and reference image list 1, respectively. W0pass2 and MV1pass2 are the motion vector refined by the second step of multi-step DMVR, MVfirstPass is the motion refinement derived in the first step of optical flow-based motion refinement. sbIdx3 is the sub-block index for the first step of optical flow-based motion refinement. sbIdx2 is the sub-block index for the second stage of multi-stage DMVR.

[0143] After the first stage of optical flow-based motion vector refinement, for each sub-block 8^8, based on MV 0OFpass1 and M71OFpass1, which is obtained in the first stage of optical flow-based motion refinement, two predicted blocks Pred0_sb8ext and Pred1_sb8ext are obtained by interprediction, where Pred0_sb8ext is the predicted block from list 0 of the reference image using MV0OFpass1 and Pred1_sb8ext is the predicted block from list 1 of the reference image using W1Ofpass1. In the interprediction, the interpolation filter could be applied to generate the predicted pixel values ​​at the sub-block positions. 8-touch, 12-touch, or other interpolation filters can be used. Since gradients are needed in the optical flow equations, the predicted block is then extended. In this configuration, shown in FIG. 13A, the 3-pixel rows and 3-pixel columns are extended to the four edges of the 8x8 sub-block.In some other modes, in order to reduce bandwidth and interpolation calculation, the extended pixels are filled from the boundary pixels within the 8x8 sub-block, instead of actually performing interpolation on them, as shown in FIG. 5. Petition 870250092849, dated 10 / 10 / 2025, pp. 64 / 119 57 / 77 In some modes, to reduce calculation, the refined motion vectors of two surrounding 8*8 sub-blocks are checked. If two surrounding 8*8 sub-blocks have the same refined motion vectors My0OFpass1 and W10Fpass1, the two surrounding 8*8 blocks can be merged into a 16*8 or 8*16 sub-block. Then, the motion vector of the merged sub-block is continuously compared to the refined motion vectors of its surrounding 8*8 sub-block. If the motion vectors are still the same, then the two sub-blocks are further merged. The merging process is repeated to find the largest possible sub-block. Then, the interpretation is performed on the merged sub-block. Assuming that the second stage of optical flow-based motion refinement is performed at the L*L sub-block level, in some other modes, interprediction could also be performed on each L*L sub-block within the 8*8 sub-block separately.This will not affect the results.

[0144] After the two predicted blocks are obtained, the optical flow-based motion vector refinement method can be applied to each L*L sub-block in the 8*8 sub-block, where L is a positive integer equal to or less than 8. For example, equations (1) to (4) or equations (7) to (11) are used to derive the motion refinement MVsecondFass= (yx_secondPass, vy_secondPass) for the second stage of optical flow-based motion refinement. Similar to BDOF, the current SAD or SATD between Pred0_sbL and Pred1_sbL can be calculated and compared to a threshold, where Pred0_sbL is the predicted sub-block of the current L*L sub-block from list 0 of the reference image, which is a sub-block of Pred0_sb8ext, and Pred1_sb4 is the predicted sub-block of the current L*L sub-block from list 1 of the reference image, which is a sub-block of Pred1_sb8ext. If the SAD is less than the threshold, the refinement process is skipped. Petition 870250092849, dated 10 / 10 / 2025, pp. 65 / 119 58 / 77 this LxL subblock. The boundary, for example, can be defined as kxsbWidthxsbHeight, where sbWidth and sbHeight are the width and height of the predicted subblock for the SAD or SATD calculation, and k is a positive factor. As shown in FIG. 13B, for an LxL subblock, the pixels in the extended (L+6)x(L+6) area are needed to derive the motion refinement WsecondPassathrough equations (1) to (4) or equations (7) to (11). After deriving the motion refinement from the second step of the optical flow-based refinement, the motion vector of each LxL subblock can be derived as: • My0OFpass2(sW%4) = MV0OFpassl(sbIdx3) + MVsecondPass(sbIdx4) (23) • MyioFpaSS2(sb / dx4) = MV1oFpaSSi(sbIdx3) - MVSecondPaSS(sbIdx4) where W0Ofpass2 and W1OPpass2 are the motion vector refined by the second optical flow-based refinement step for reference image list 0 and reference image list 1, respectively. where M70OFpass1 and MV1OFpass1 are the motion vectors refined by the first optical flow-based refinement step for reference image list 0 and reference image list 1, respectively. MysecondPass is the motion refinement derived from the second optical flow-based motion refinement step. sbIdx4 is the sub-block index for the second stage of optical flow-based motion refinement, and sbIdx3 is the sub-block index for the first stage of optical flow-based motion refinement.

[0145] The MV0OFpass2e W0Ofpass1 derivatives are the final motion vectors for the current LxL sub-block. After that, existing motion compensation could be performed. Sub-block based BDOF or sample-based BDOF could also be used in motion compensation.

[0146] In the modes described above, the proposed multi-step optical flow-based motion refinement follows the two Petition 870250092849, dated 10 / 10 / 2025, pp. 66 / 119 59 / 77 first stages of the multi-stage DMVR. By doing so, the multi-stage DMVR is extended to four stages. However, the proposed method is not necessarily combined with the multi-stage DMVR. It can be used independently of the DMVR. For a bi-predicted CU, if certain conditions are met, the proposed multi-stage optical flow-based motion refinement can be performed to refine the CU's motion. Thus, the input of the proposed multi-stage optical flow-based motion refinement can be the motion vector derived in merge mode that is inherited from previous encoded blocks or decoded from the bitstream. And the output of the proposed multi-stage optical flow-based motion refinement can be further refined by other coding tools before motion compensation. The condition for multi-stage optical flow-based motion refinement can be, but is limited to: - The CU is encoded using "true" bi-prediction mode, meaning that one of the two reference images precedes the current image in the display order, and the other follows the current image in the display order. The distances (i.e., POC difference) from the two reference images to the current image are equal. Both reference images are short-term reference images. - The CU is not encoded using affine mode or SbTMVP blending mode, local luminance compensated prediction mode, or overlapping block motion compensation. CU has more than 64 luma samples. Both the height and width of the CU are greater than or equal to 8 luma samples. - Biprediction with weight index of weight at CU level Petition 870250092849, dated 10 / 10 / 2025, page 67 / 119 60 / 77 (BCW) indicates equal weight. Weighted Prediction (WP) is not enabled for the current CU. The combined interprediction and intraprediction (CIIP) mode is not used for the current CU. - The symmetrical motion vector difference is not used - The CU is not encoded as a merge mode with the motion vector difference.

[0147] In some modalities, the difference between the two predicted blocks of the current CU, one from reference image list 0 and the other from reference image list 1, is calculated and based on the difference, optical flow-based refinement is ignored. The difference can be the sum of absolute differences (SAD) or the sum of absolute differences transformed (SATD) of the two predicted blocks.

[0148] In one example, the number of refinement steps is dependent on the difference between the two predicted blocks. If the difference is less than or equal to a threshold, only one optical flow-based refinement step is performed (the second optical flow-based refinement step is skipped); otherwise, two optical flow-based refinement steps are performed. Or if the difference is less than or equal to a first threshold, no optical flow-based refinement is performed; if the difference is greater than the first threshold but less than or equal to a second threshold, only one optical flow-based refinement step is performed; if the difference is greater than the second threshold, three optical flow-based refinement steps are performed.

[0149] In another example, depending on the difference between the two predicted blocks, optical flow-based refinement is either ignored or not. First, the two predicted blocks are interpolated using Petition 870250092849, dated 10 / 10 / 2025, pp. 68 / 119 61 / 77 motion vector before refinement (indicated as MVi) and the difference between these predicted blocks is calculated, indicated as Do. If Do is less than or equal to a first threshold, no optical flow-based refinement is performed; if Do is greater than a threshold, the first optical flow-based refinement step is performed and the refined motion vector, indicated as MVi, is obtained. The other two predicted blocks are obtained using the refined motion vector MVi, and the difference between these two predicted blocks is calculated, indicated as Di, and compared to a second threshold. If Di is equal to or less than a second threshold, the refinement process ends; if Do is greater than a threshold, the second optical flow-based refinement step is performed continuously, and the motion vector is refined again.

[0150] In yet another example, the decision to perform optical flow-based refinement is made based on the change in the difference between two predicted blocks. First, the two predicted blocks are interpolated using a motion vector before refinement (denoted as MVo), and the difference between these predicted blocks is calculated, denoted as Do. Then, optical flow-based refinement is performed, and the motion vector is refined, denoted as MVi. Then, two other predicted blocks are obtained using the refined motion vector MV1, and the difference between these two predicted blocks is calculated, denoted as Di. If Di is greater than Do, the results of the optical flow-based refinement are reversed.In other words, the refined motion vector MV1 is replaced by the original motion vector MVo; if Di is less than or equal to Do, the result of the optical flow-based refinement is maintained, and the next process is executed based on the refinement motion vector MVi. In some other examples, the difference changes are used to decide whether to execute another optical flow-based refinement step. If... Petition 870250092849, dated 10 / 10 / 2025, pp. 69 / 119 62 / 77 If Di is greater than Do, the refinement process is terminated and the refinement motion vector MVi is sent to the next step; if Di is less than or equal to Do, another optical flow-based refinement step is performed based on MVi to obtain another refined motion vector MV2. Thus, the number of optical flow-based refinement steps is based on the difference of differences between two predicted blocks before and after refinement.

[0151] Variants of the embodiments described above can be developed to implement the optical flow-based motion refinement method. In some embodiments, there may be more than two optical flow-based motion refinement steps for a CU. For example, the first step is performed at the 8^8 sub-block level, the second step is performed at the 4x4 sub-block level, and the third step is performed at the 2x2 sub-block level. As another example, the first step is performed at the 8x8 sub-block level, the second step is performed at the 8x8 sub-block level, and the third step is also performed at the 8x8 sub-block level.

[0152] In some modalities, the number of optical flow-based motion refinement steps depends on the CU size, sequence resolution, or quantization parameters. For example, for large CUs, there are few steps, and for small CUs, there are many steps, since large CUs generally have smoother content that can be refined on a large mesh, and multiple optical flow-based motion refinement steps may not be necessary. For example, there is only one step for a CU having an area greater than or equal to 4096; there are two steps for CUs having an area less than 4096 but greater than 256; there are three steps for CUs having an area less than 256. In another example, for CUs encoded with large param Petition 870250092849, dated 10 / 10 / 2025, pp. 70 / 119 63 / 77 For large quantization parameters, there are few steps, and for CUs encoded with small quantization parameters, there are more steps, since the large quantization parameter results in more distortion, requiring more refinement steps. In yet another example, there are few steps for large video sequences, and there are more steps for small video sequences, because, generally, there are more large CUs in large video sequences that do not need more refinement steps, and there are more small CUs in small video sequences, and they need more refinement steps.

[0153] In some modes, the sub-block size for optical flow-based motion refinement is dependent on the CU size, the quantization parameter, or the resolution of the video sequence. For example, for a large CU, optical flow-based motion refinement is performed at a large sub-block level, and for a small CU, optical flow-based motion refinement is performed at a small sub-block level. For example, for a CU greater than 256, optical flow-based refinement is performed at the 8x8 sub-block level, and for a CU less than or equal to 256, optical flow-based refinement is performed at the 4x4 sub-block level.For another example, for CUs encoded with large quantization parameters, optical flow-based motion refinement is performed at a large sub-block level, and for CUs encoded with small quantization parameters, optical flow-based motion refinement is performed at a small sub-block level. In yet another example, for large video sequences, optical flow-based motion refinement is performed at a large sub-block level, and for small video sequences, optical flow-based motion refinement is performed at a small sub-block level. Petition 870250092849, dated 10 / 10 / 2025, pp. 71 / 119 64 / 77

[0154] And the adaptive sub-block size for optical flow-based motion refinement can be combined with multi-step optical flow-based motion refinement. For example, for CUs having an area greater than 1024, the first step of optical flow-based motion refinement is performed at the 16*16 sub-block level and the second step is performed at the 8*8 sub-block level; for CUs having an area less than or equal to 1024, the first step of optical flow-based motion refinement is performed at the 8*8 sub-block level and the second step is performed at the 4*4 sub-block level.For another example, for CUs with an area greater than or equal to 1024, both steps are executed at the 8*8 sub-block level; for CUs with an area less than 1024 but greater than or equal to 512, the first step is executed at the 8*8 sub-block level and the second step is executed at the 4*4 sub-block level; and for CUs with an area less than 512, both steps are executed at the 4*4 sub-block level. For yet another example, for CUs encoded with quantization parameters greater than or equal to 32, the first step of optical flow-based motion refinement is executed at the 16*16 sub-block level and the second step is executed at the 8*8 sub-block level; For CUs encoded with a quantization parameter less than 32, the first optical flow-based motion refinement step is performed at the 8*8 sub-block level, and the second step is performed at the 4*4 sub-block level.For yet another example, for video sequences with a resolution greater than 1920*1080, the first stage of optical flow-based motion refinement is performed at the 16*16 sub-block level and the second stage is performed at the 8*8 sub-block level; for video sequences with a resolution less than 1920*1080, the first stage of optical flow-based motion refinement is performed at the 8*8 sub-block level and the second stage is performed at the 4*4 sub-block level. Petition 870250092849, dated 10 / 10 / 2025, pp. 72 / 119 65 / 77

[0155] Furthermore, the adaptive sub-block size for optical flow-based motion refinement can be combined with optical flow-based motion refinement with an adaptive number of steps. For example, for CUs whose area is greater than 1024, there is only one optical flow-based motion refinement step that is performed at the 16*16 sub-block level; for CUs whose area is less than or equal to 1024, there are two optical flow-based motion refinement steps, the first of which is performed at the 8*8 sub-block level and the second step is performed at the 4*4 sub-block level.As another example, for CUs encoded with quantization parameters greater than or equal to 32, there are two optical flow-based motion refinement steps, the first of which is performed at the 16*16 sub-block level and the second at the 8*8 sub-block level; for CUs encoded with a quantization parameter less than 32, there is only one optical flow-based motion refinement step, which is performed at the 8*8 sub-block level. As yet another example, for a video sequence with a resolution greater than 1920*1080, there is only one optical flow-based motion refinement step, which is performed at the 16*16 sub-block level. For a sequence of videos with a resolution lower than 1920*1080, there are two stages of optical flow-based motion refinement, the first stage being performed at the 8*8 sub-block level and the second stage at the 4*4 sub-block level.

[0156] The embodiments described in the present invention can be freely combined.

[0157] In some embodiments, a non-transient, computer-readable storage medium is also provided that stores a bitstream. The bitstream can be encoded and decoded. Petition 870250092849, dated 10 / 10 / 2025, pp. 73 / 119 66 / 77 according to the published method of optical flow-based motion refinement.

[0158] In some embodiments, a non-transient computer-readable storage medium containing instructions is also provided, and the instructions can be executed by a device (such as the disclosed encoder and decoder) to perform the methods described above. Common forms of non-transient media include, for example, a floppy disk, a floppy disk, a hard disk, a solid-state drive, magnetic tape or any other magnetic data storage media, a CD-ROM, any other optical data storage media, any other physical media with hole patterns, a RAM, a PROM and EPROM, a FLASH-EPROM or any other flash memory, NVRAM, a cache, a register, any other memory chip or cartridge, and network versions thereof. The device may include one or more processors (CPUs), an input / output interface, a network interface and / or memory.

[0159] The modalities can be further described using the following clauses: 1. Video processing method comprising: dividing a coding block into a first set of sub-blocks and a second set of sub-blocks; Perform a first step of optical flow-based motion vector refinement on the first set of sub-blocks; and perform a second step of optical flow-based motion vector refinement on the second set of sub-blocks. 2. Method, according to clause 1, in which the sizes of the first set of sub-blocks are different from the sizes of the second set of sub-blocks. 3. Method, according to clause 2, additionally comprising: Petition 870250092849, dated 10 / 10 / 2025, pp. 74 / 119 67 / 77 divide the coding block into a third set of subblocks; and perform a third step of optical flow-based motion vector refinement on the third set of subblocks. 4. Method, according to clause 2, where: Each of the first set of sub-blocks has a size of 2x2, 4x4, 8x8, or 16x16; Each of the second set of sub-blocks has a size of 2x2, 4x4, 8x8, or 16x16; and each of the third set of sub-blocks has a size of 2x2, 4x4, 8x8, or 16x16. 5. Method, according to clause 2, in which the optical flow-based motion vector refinement is bidirectional. 6. Method, according to clause 2, additionally comprising: Before the first, second, and third optical flow-based motion vector refinement steps are executed, perform one or more decoder-facing motion vector refinement (DMVR) steps on the encoding block. 7. Method, according to clause 6, in which executing one or more DMVR steps in the coding block comprises: Perform an initial refinement step of the bilateral matching motion vector in the coding block; Divide the coding block into a fourth set of subblocks; and perform a second step of bilateral matching motion vector refinement on the fourth set of subblocks. 8. Method, according to clause 2, in which the first, second and third steps of optical flow-based motion vector refinement belong to a number of refinement steps. Petition 870250092849, dated 10 / 10 / 2025, pp. 75 / 119 68 / 77 optical flow-based motion vector executed in the encoding block, and the method further comprises: Determine the number of steps based on a coding block size. 9. Method, according to clause 2, additionally comprising: To determine, based on a predetermined condition being met, a sub-block size for each of the first set of sub-blocks, the second set of sub-blocks, or the third set of sub-blocks. 10. Method, according to clause 9, in which the predetermined condition comprises at least one of: The encoding block is encoded using a biprediction mode; The encoding block has a size that exceeds a predetermined limit; The encoding block is smaller than a predetermined limit; Weighted prediction is disabled for the coding block; Combined interprediction and intraprediction (CIIP) is disabled for the coding block; Local luma compensation is enabled for the encoding block; The sub-block motion compensation is applied to the encoding block; The symmetrical motion vector difference is not used; or the coding block is not coded as a merge mode with the motion vector difference. 11. Method, according to clause 1, comprising Petition 870250092849, dated 10 / 10 / 2025, pp. 76 / 119 69 / 77 additionally: Generate interprediction samples based on a refined motion vector; and encode a bitstream based on the interprediction samples. 12. Method, according to clause 1, additionally comprising: To decode a bitstream associated with a video; and to reconstruct the encoding block. 13. Apparatus, comprising: a memory that stores computer instructions; and one or more processors configured to execute the computer instructions, wherein the execution of the computer instruction causes the apparatus to perform the operations comprising: to divide a coding block into a first set of sub-blocks and a second set of sub-blocks; Perform a first step of optical flow-based motion vector refinement on the first set of sub-blocks; and perform a second step of optical flow-based motion vector refinement on the second set of sub-blocks. 14. Device, according to clause 13, in which the sizes of the first set of sub-blocks are different from the sizes of the second set of sub-blocks. 15. Device, according to clause 14, in which the execution of the computer instruction causes the device to additionally execute: divide the encoding block into a third set of subblocks; and perform a third step of optical flow-based motion vector refinement on the third set of subblocks. Petition 870250092849, dated 10 / 10 / 2025, pp. 77 / 119 70 / 77 16. Device, in accordance with clause 14, wherein: Each of the first set of sub-blocks has a size of 2x2, 4x4, 8x8, or 16x16; Each of the second set of sub-blocks has a size of 2x2, 4x4, 8x8, or 16x16; and each of the third set of sub-blocks has a size of 2x2, 4x4, 8x8, or 16x16. 17. Apparatus, according to clause 14, in which the motion vector refinement based on optical flow is bidirectional. 18. Device, according to clause 14, in which the execution of the computer instruction causes the device to additionally execute: Before the first, second, and third optical flow-based motion vector refinement steps are executed, perform one or more decoder-facing motion vector refinement (DMVR) steps on the encoding block. 19. Device, according to clause 18, in which, when executing one or more DMVR steps in the encoding block, the execution of the computer instruction causes the device to additionally execute: Perform an initial refinement step of the bilateral matching motion vector in the coding block; Divide the coding block into a fourth set of subblocks; and perform a second step of bilateral matching motion vector refinement on the fourth set of subblocks. 20. Apparatus, according to clause 14, in which the first, second and third optical flow-based motion vector refinement steps belong to a number of optical flow-based motion vector refinement steps performed on Petition 870250092849, dated 10 / 10 / 2025, pp. 78 / 119 71 / 77 coding block, and the execution of the computer instruction causes the device to additionally execute: Determine the number of steps based on a coding block size. 21. Device, according to clause 14, in which the execution of the computer instruction causes the device to additionally execute: To determine, based on a predetermined condition being met, a sub-block size for each of the first set of sub-blocks, the second set of sub-blocks, or the third set of sub-blocks. 22. Device, in accordance with clause 21, where the predetermined condition includes at least one of: The encoding block is encoded using a biprediction mode; The encoding block has a size that exceeds a predetermined limit; The encoding block is smaller than a predetermined limit; Weighted prediction is disabled for the coding block; Combined interprediction and intraprediction (CIIP) is disabled for the coding block; Local luma compensation is enabled for the encoding block; The sub-block motion compensation is applied to the encoding block; The symmetrical motion vector difference is not used; or the coding block is not coded as a merge mode with the motion vector difference. Petition 870250092849, dated 10 / 10 / 2025, pp. 79 / 119 72 / 77 23. Device, according to clause 13, in which the execution of the computer instruction causes the device to additionally execute: Generate interprediction samples based on a refined motion vector; and encode a bitstream based on the interprediction samples. 24. Device, according to clause 13, in which the execution of the computer instruction causes the device to additionally execute: To decode a bitstream associated with a video; and to reconstruct the encoding block. 25. Non-transient, computer-readable storage media that stores a bitstream of a video for processing, according to a method comprising: to divide a coding block into a first set of sub-blocks and a second set of sub-blocks; Perform a first step of optical flow-based motion vector refinement on the first set of sub-blocks; and perform a second step of optical flow-based motion vector refinement on the second set of sub-blocks. 26. Non-transient, computer-readable storage media, according to clause 25, wherein the sizes of the first set of sub-blocks are different from the sizes of the second set of sub-blocks. 27. Non-transient, computer-readable storage media, in accordance with clause 26, wherein the method further comprises: divide the coding block into a third set of subblocks; and Petition 870250092849, dated 10 / 10 / 2025, pp. 80 / 119 73 / 77 perform a third step of optical flow-based motion vector refinement on the third set of sub-blocks. 28. Non-transient, computer-readable storage media, in accordance with clause 26, wherein: Each of the first set of sub-blocks has a size of 2x2, 4x4, 8x8, or 16x16; Each of the second set of sub-blocks has a size of 2x2, 4x4, 8x8, or 16x16; and each of the third set of sub-blocks has a size of 2x2, 4x4, 8x8, or 16x16. 29. Non-transient, computer-readable storage media, in accordance with clause 26, wherein optical flow-based motion vector refinement is bidirectional. 30. Method, according to clause 26, wherein the method additionally comprises: Before the first, second, and third optical flow-based motion vector refinement steps are executed, perform one or more decoder-facing motion vector refinement (DMVR) steps on the encoding block. 31. Non-transient, computer-readable storage media, as defined in clause 30, wherein performing one or more DMVR steps on the encoding block comprises: Perform an initial refinement step of the bilateral matching motion vector in the coding block; Divide the coding block into a fourth set of subblocks; and perform a second step of bilateral matching motion vector refinement on the fourth set of subblocks. 32. Non-transient, computer-readable storage media, in accordance with clause 26, wherein the first, second and Petition 870250092849, dated 10 / 10 / 2025, pp. 81 / 119 74 / 77 The third optical flow-based motion vector refinement steps belong to a number of optical flow-based motion vector refinement steps performed in the coding block, and the method additionally comprises: Determine the number of steps based on a coding block size. 33. Non-transient, computer-readable storage media, in accordance with clause 26, wherein the method further comprises: To determine, based on a predetermined condition being met, a sub-block size for each of the first set of sub-blocks, the second set of sub-blocks, or the third set of sub-blocks. 34. Non-transient, computer-readable storage media, in accordance with clause 33, wherein the predetermined condition comprises at least one of: The encoding block is encoded using a biprediction mode; The encoding block has a size that exceeds a predetermined limit; The encoding block is smaller than a predetermined limit; Weighted prediction is disabled for the coding block; Combined interprediction and intraprediction (CIIP) is disabled for the coding block; Local luma compensation is enabled for the encoding block; The sub-block motion compensation is applied to the encoding block; Petition 870250092849, dated 10 / 10 / 2025, page 82 / 119 75 / 77 the symmetrical motion vector difference is not used; or the coding block is not coded as a merge mode with the motion vector difference. 35. Non-transient, computer-readable storage media, in accordance with clause 25, wherein the method further comprises: Generate interprediction samples based on a refined motion vector; and encode a bitstream based on the interprediction samples. 36. Non-transient, computer-readable storage media, in accordance with clause 25, wherein the method further comprises: Decode a bitstream associated with the video; and reconstruct the encoding block.

[0160] It should be noted that relational terms in this document, such as “first” and “second,” are used only to differentiate one entity or operation from another entity or operation, and do not require or imply any actual relationship or sequence between those entities or operations. Furthermore, the words “comprising,” “having,” “containing,” and “including,” and other similar forms are intended to be equivalent in meaning and to be open-ended, so that an item or items following any of these words do not signify a comprehensive list of such item or items, or be limited only to the item or items listed.

[0161] As used in this document, except where specifically stated otherwise, the term “or” encompasses all possible combinations, except where impractical. For example, if it is stated that a database may include A or B, then, except where specifically stated otherwise or impractical, the database Petition 870250092849, dated 10 / 10 / 2025, p. 83 / 119 76 / 77 data may include A, or B, or A and B. As per the second example, if it is stated that a database may include A, B, or C, then, unless specifically stated otherwise or if it is impractical, the database may include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0162] It is recognized that the embodiments described above can be implemented by hardware, software (program codes), or a combination of hardware and software. If implemented by software, they can be stored on the computer-readable media described above. The software, when executed by the processor, can execute the disclosed methods. The computing units and other functional units described in the present invention can be implemented by hardware, or software, or a combination of hardware and software. A person skilled in the art will also understand that multiple modules / units described above can be combined as one module / unit, and each of the modules / units described above can be further divided into a plurality of submodules / subunits.

[0163] In the preceding descriptive report, embodiments were described with reference to various specific details that may vary from implementation to implementation. Certain adaptations and modifications of the described embodiments may be made. Other embodiments may be apparent to those skilled in the art from consideration of the descriptive report and practice of the invention disclosed in this document. The descriptive report and examples are intended to be considered only as illustrative, with the true scope and spirit of the invention being indicated by the following claims. It is also intended that the sequence of steps shown in the figures is for illustrative purposes only and is not limited to any particular sequence of steps. Thus, those skilled in the art may Petition 870250092849, dated 10 / 10 / 2025, pp. 84 / 119 77 / 77 understand that these steps can be performed in a different order while implementing the same method.

[0164] In the drawings and descriptive report, exemplary modalities were disclosed. However, many variations and modifications can be made to these modalities. Consequently, although specific terms are employed, they are used only in a generic and descriptive sense and not for the purpose of limitation. Petition 870250092849, dated 10 / 10 / 2025, page 85 / 119

Claims

1 / 5 CLAIMS 1. A video processing method, characterized in that it comprises: dividing a coding block into a first set of sub-blocks and a second set of sub-blocks; performing a first stage of optical flow-based motion vector refinement on the first set of sub-blocks; and performing a second stage of optical flow-based motion vector refinement on the second set of sub-blocks.

2. Method according to claim 1, characterized in that the sizes of the first set of sub-blocks are different from the sizes of the second set of sub-blocks.

3. A method according to claim 2, characterized in that it further comprises: dividing the coding block into a third set of subblocks; and performing a third optical flow-based motion vector refinement step on the third set of subblocks.

4. Method according to claim 2, characterized in that: each of the first set of sub-blocks has a size of 2^2, 4x4, 8x8 or 16x16; each of the second set of sub-blocks has a size of 2x2, 4x4, 8x8 or 16x16; and each of the third set of sub-blocks has a size of 2x2, 4x4, 8x8 or 16x16.

5. Method according to claim 2, characterized in that the optical flow-based motion vector refinement is bidirectional.

6. Method according to claim 2, characterized in Petition 870250092849, dated 10 / 10 / 2025, page 86 / 119 2 / 5, in that it further comprises: before the first, second and third optical flow-based motion vector refinement steps are performed, performing one or more decoder-oriented motion vector refinement (DMVR) steps on the encoding block.

7. A method according to claim 6, characterized in that performing one or more DMVR steps on the coding block comprises: performing a first step of bilateral matching motion vector refinement on the coding block; dividing the coding block into a fourth set of subblocks; and performing a second step of bilateral matching motion vector refinement on the fourth set of subblocks.

8. A method according to claim 2, characterized in that the first, second, and third optical flow-based motion vector refinement steps belong to a number of optical flow-based motion vector refinement steps performed on the coding block, and the method further comprises: determining the number of steps based on a coding block size.

9. Method according to claim 2, characterized in that it further comprises: determining, based on a predetermined condition being satisfied, a sub-block size for each of the first set of sub-blocks, the second set of sub-blocks, or the third set of sub-blocks.

10. Method according to claim 9, characterized in that the predetermined condition comprises at least one of: the coding block is coded using a biprediction mode; the coding block has a size that exceeds a predetermined limit; the coding block has a size smaller than a predetermined limit; weighted prediction is disabled for the coding block; combined interprediction and intraprediction (CIIP) is disabled for the coding block; local luma compensation is enabled for the coding block; sub-block motion compensation is applied to the coding block; symmetric motion vector difference is not used; or the coding block is not coded as a merge mode with motion vector difference.

11. A method according to claim 1, characterized in that it further comprises: generating interprediction samples based on a refined motion vector; and encoding a bitstream based on the interprediction samples.

12. Method according to claim 1, characterized in that it further comprises: decoding a bitstream associated with a video; and reconstructing the encoding block.

13. Apparatus, characterized in that it comprises: a memory that stores computer instructions; and Petition 870250092849, of 10 / 10 / 2025, page 88 / 119 4 / 5 one or more processors configured to execute computer instructions, wherein the execution of the computer instruction causes the apparatus to perform operations comprising: dividing a coding block into a first set of sub-blocks and a second set of sub-blocks; performing a first step of optical flow-based motion vector refinement on the first set of sub-blocks; and performing a second step of optical flow-based motion vector refinement on the second set of sub-blocks.

14. Apparatus, according to claim 13, characterized in that the sizes of the first set of sub-blocks are different from the sizes of the second set of sub-blocks.

15. Apparatus, according to claim 14, characterized in that the execution of the computer instruction causes the apparatus to additionally perform: dividing the coding block into a third set of subblocks; and performing a third optical flow-based motion vector refinement step on the third set of subblocks.

16. Apparatus, according to claim 14, characterized in that: each of the first set of sub-blocks has a size of 2*2, 4*4, 8*8 or 16*16; each of the second set of sub-blocks has a size of 2*2, 4*4, 8*8 or 16*16; and each of the third set of sub-blocks has a size of 2*2, 4*4, 8*8 or 16*16.

17. Apparatus, according to claim 14, characterized in that the optical flow-based motion vector refinement is bidirectional. Petition 870250092849, dated 10 / 10 / 2025, p. 89 / 119 5 / 5 18. Apparatus, according to claim 14, characterized in that the execution of the computer instruction causes the apparatus to additionally execute: before the first, second and third optical flow-based motion vector refinement steps are executed, execute one or more decoder-oriented motion vector refinement (DMVR) steps in the encoding block.

19. Device according to claim 18, characterized in that when executing one or more DMVR steps in the coding block, the execution of the computer instruction causes the device to additionally perform: a first bilateral matching motion vector refinement step in the coding block; dividing the coding block into a fourth set of subblocks; and performing a second bilateral matching motion vector refinement step in the fourth set of subblocks.

20. Non-transient computer-readable storage media characterized in that it stores a bitstream of video for processing according to a method comprising: dividing a coding block into a first set of sub-blocks and a second set of sub-blocks; performing a first optical flow-based motion vector refinement step on the first set of sub-blocks; and performing a second optical flow-based motion vector refinement step on the second set of sub-blocks. Petition 870250092849, dated 10 / 10 / 2025, pp. 90 / 119

Images