Interaction between different decoder-side motion vector derivation modes

Abstract

An apparatus and method for video processing involving interaction between different decoder-side motion vector derivation modes are described. An example method for video processing includes: determining unupdated motion information associated with a current block; updating the unupdated motion information based on multiple decoder-side motion vector derivation methods to generate updated motion information for the current block; and performing a conversion between the current block and a bitstream representation of the video including the current block based on the updated motion information.

Description

[0001] This divisional application is filed on August 5, 2019, with application number 201910718733.1 and invention title "Interaction between different decoder-side motion vector derivation modes".

[0002] Cross-reference to related applications

[0003] In accordance with applicable patent law and / or the Paris Convention, this application promptly claims priority and interest in International Patent Application No. PCT / CN2018 / 098691, filed August 4, 2018, and International Patent Application No. PCT / CN2018 / 109250, filed October 6, 2018. The entire disclosure of International Patent Application Nos. PCT / CN2018 / 098691 and PCT / CN2018 / 109250 is incorporated herein by reference as a part of this application disclosure. Technical Field

[0004] This patent document relates to video coding technologies, devices, and systems. Background Technology

[0005] Despite advancements in video compression, digital video still accounts for the largest share of bandwidth usage on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth required for digital video usage is expected to continue to grow. Summary of the Invention

[0006] Devices, systems, and methods related to digital video coding are described, and specifically, motion refinement based on updated motion vectors generated from two-step inter-frame prediction is described. The described methods can be applied to existing video coding standards (e.g., High Efficiency Video Coding (HEVC)) and future video coding standards or video codecs.

[0007] In one representative aspect, a video processing method is provided, comprising: determining original motion information of a current block; scaling an original motion vector and a derived motion vector derived from the original motion vector to the same target precision; generating an updated motion vector from the scaled original and derived motion vectors; and performing a conversion between the current block and a bitstream representation of the video including the current block based on the updated motion vector.

[0008] In another representative aspect, a video processing method is provided, comprising: determining the original motion information of a current block; updating the original motion vector of the original motion information of the current block based on a thinning method; cropping the updated motion vector to a range; and performing a conversion between the current block and a bitstream representation of the video including the current block based on the cropped updated motion vector.

[0009] In another representative aspect, a video processing method is provided, comprising: determining raw motion information associated with a current block; generating updated motion information based on a specific prediction mode; and performing a conversion between the current block and a bitstream representation of video data including the current block based on the updated motion information, wherein the specific prediction mode includes one or more of bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC) techniques, or template matching techniques.

[0010] In another representative aspect, a video processing method is provided, comprising: determining the MVD precision of a current block processed in an affine mode from a set of motion vector difference (MVD) precisions; and performing a conversion between the current block and a bitstream representation of the video including the current block based on the determined MVD precision.

[0011] In another representative aspect, a video processing method is provided, comprising: determining unupdated motion information associated with a current block; updating the unupdated motion information based on a multiple decoder-side motion vector derivation (DMVD) method to generate updated motion information for the current block; and performing a conversion between the current block and a bitstream representation of the video including the current block based on the updated motion information.

[0012] In another representative aspect, the disclosed technology can be used to provide a method for video coding. This method includes receiving a bitstream representation of a current block of video data; generating updated first and second reference motion vectors based on weighted sums of a first scaled motion vector and first and second scaled reference motion vectors, respectively; wherein the first motion vector is derived based on a first reference motion vector from a first reference block and a second reference motion vector from a second reference block; wherein the current block is associated with the first and second reference blocks; wherein the first scaled motion vector is generated by scaling the first motion vector to a target precision; and wherein the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to the target precision, respectively; and processing the bitstream representation based on the updated first and second reference motion vectors to generate the current block.

[0013] In another representative aspect, the disclosed technology can be used to provide a method for video coding. This method includes generating an intermediate prediction for a current block based on first motion information associated with the current block, updating the first motion information to second motion information, and generating a final prediction for the current block based on the intermediate prediction or the second motion information.

[0014] In another representative aspect, the disclosed technology can be used to provide a method for video coding. The method includes receiving a bitstream representation of a current block of video data; generating intermediate motion information based on motion information associated with the current block; generating updated first and second reference motion vectors based on first and second reference motion vectors, respectively, wherein the current block is associated with first and second reference blocks, and wherein the first and second reference motion vectors are associated with first and second reference blocks, respectively; and processing the bitstream representation based on the intermediate motion information or the updated first and second reference motion vectors to generate the current block.

[0015] In another representative aspect, the disclosed technology can be used to provide a method for video coding. This method includes generating an intermediate prediction for a current block based on first motion information associated with the current block, updating the first motion information to second motion information, and generating a final prediction for the current block based on the intermediate prediction or the second motion information.

[0016] In another representative aspect, the disclosed technology can be used to provide a method for video coding. The method includes receiving a bitstream representation of a current block of video data; generating intermediate motion information based on motion information associated with the current block; generating updated first and second reference motion vectors based on first and second reference motion vectors, respectively, wherein the current block is associated with first and second reference blocks, and wherein the first and second reference motion vectors are associated with first and second reference blocks, respectively; and processing the bitstream representation based on the intermediate motion information or the updated first and second reference motion vectors to generate the current block.

[0017] In another representative aspect, the disclosed technology can be used to provide a method for video coding. This method includes generating an intermediate prediction for a current block based on first motion information associated with the current block, updating the first motion information to second motion information, and generating a final prediction for the current block based on the intermediate prediction or the second motion information.

[0018] In another representative aspect, the disclosed technology can be used to provide a method for video coding. This method includes receiving a bitstream representation of a current block of video data, generating intermediate motion information based on motion information associated with the current block, generating updated first and second reference motion vectors based on first and second reference motion vectors respectively, wherein the current block is associated with first and second reference blocks, and wherein the first and second reference motion vectors are associated with first and second reference blocks respectively, and processing the bitstream representation based on the intermediate motion information or the updated first and second reference motion vectors to generate the current block.

[0019] In another representative aspect, the disclosed technology can be used to provide a method for video coding, which includes generating an updated reference block for the bitstream representation of the current block by modifying a reference block associated with the current block; calculating a temporal gradient for bidirectional optical flow (BIO) motion refinement based on the updated reference block; and performing a transformation including BIO motion refinement between the bitstream representation and the current block based on the temporal gradient.

[0020] In another representative aspect, the disclosed technology can be used to provide a method for video coding, the method comprising: generating a temporal gradient for bidirectional optical flow (BIO) motion refinement for a bitstream representation of a current block; generating an updated temporal gradient by subtracting the difference between a first mean and a second mean from the temporal gradient, wherein the first mean is the mean of a first reference block, wherein the second mean is the mean of a second reference block, and wherein the first and second reference blocks are associated with the current block; and performing a transformation including BIO motion refinement between the bitstream representation and the current block based on the updated temporal gradient.

[0021] In another representative aspect, the above methods are embodied in processor-executable code and stored in a computer-readable program medium.

[0022] In another representative aspect, a device configured or operable to perform the methods described above is disclosed. This device may include a processor programmed to implement the methods.

[0023] In another representative aspect, video decoder devices can implement the methods described in this paper.

[0024] The above and other aspects and features of the disclosed technology are described in more detail in the accompanying drawings, description and claims. Attached Figure Description

[0025] Figure 1 An example of constructing a Merge candidate list is shown.

[0026] Figure 2 An example of a spatial candidate location is shown.

[0027] Figure 3 An example of a candidate pair that has undergone redundancy checks of spatial Merge candidates is shown.

[0028] Figure 4A and 4B An example of the position of a second prediction unit (PU) based on the size and shape of the current block is shown.

[0029] Figure 5 An example of motion vector scaling for time merge candidates is shown.

[0030] Figure 6 An example of candidate positions for the time-merge candidate is shown.

[0031] Figure 7 An example of generating bidirectional prediction Merge candidates using a combination is shown.

[0032] Figure 8 An example of constructing motion vector prediction candidates is shown.

[0033] Figure 9 An example of motion vector scaling for spatial motion vector candidates is shown.

[0034] Figure 10 An example of motion prediction using the Optional Time Motion Vector Prediction (ATMVP) algorithm for coding units (CUs) is shown.

[0035] Figure 11 An example is shown with coding units (CUs) for sub-blocks and neighboring blocks used by the Space-Time Motion Vector Prediction (STMVP) algorithm.

[0036] Figure 12A and 12B An example snapshot of a sub-block is shown when the Overlapping Block Motion Compensation (OBMC) algorithm is used.

[0037] Figure 13 An example of neighboring samples is shown for deriving the parameters of the Local Illumination Compensation (LIC) algorithm.

[0038] Figure 14 An example of a simplified affine motion model is shown.

[0039] Figure 15 An example of the affine motion vector field (MVF) for each sub-block is shown.

[0040] Figure 16 An example of motion vector prediction (MVP) for the AF_INTER affine motion pattern is shown.

[0041] Figure 17A and 17B Example candidates for the AF_MERGE affine motion mode are shown.

[0042] Figure 18 An example of bilateral matching in the Pattern Matching Motion Vector Derivation (PMMVD) mode is shown, which is a special Merge mode based on the Frame Rate Upconversion (FRUC) algorithm.

[0043] Figure 19 An example of template matching in the FRUC algorithm is shown.

[0044] Figure 20An example of one-sided motion estimation in the FRUC algorithm is shown.

[0045] Figure 21 An example of the optical flow trajectory used by the bidirectional optical flow (BIO) algorithm is shown.

[0046] Figure 22A and 22B An example snapshot is shown using the Bidirectional Optical Flow (BIO) algorithm without block expansion.

[0047] Figure 23 An example of a decoder-side motion vector refinement (DMVR) algorithm based on bilateral template matching is shown.

[0048] Figure 24 An example of a template definition used in transform coefficient context modeling is shown.

[0049] Figure 25 Different examples of motion vector scaling are shown.

[0050] Figure 26A and 26B Examples of internal and boundary sub-blocks in PU / CU are shown.

[0051] Figure 27 A flowchart illustrating an example method for video encoding based on currently disclosed techniques is shown.

[0052] Figure 28 A flowchart is shown for another example method for video encoding based on currently disclosed techniques.

[0053] Figure 29 A flowchart is shown for another example method for video encoding based on currently disclosed techniques.

[0054] Figure 30 A flowchart is shown for another example method for video encoding based on currently disclosed techniques.

[0055] Figure 31 A flowchart is shown for another example method for video encoding based on currently disclosed techniques.

[0056] Figure 32 An example of deriving motion vectors in bidirectional optical flow-based video coding is shown.

[0057] Figure 33 A flowchart is shown for another example method for video encoding based on currently disclosed techniques.

[0058] Figure 34 A flowchart is shown for another example method for video encoding based on currently disclosed techniques.

[0059] Figure 35 A flowchart is shown for another example method for video encoding based on currently disclosed techniques.

[0060] Figure 36 A flowchart is shown for another example method for video encoding based on currently disclosed techniques.

[0061] Figure 37 This is a block diagram of an example hardware platform used to implement the visual media decoding or visual media encoding techniques described in this document.

[0062] Figure 38 A flowchart is shown for another example method for video processing based on currently disclosed techniques.

[0063] Figure 39 A flowchart is shown for another example method for video processing based on currently disclosed techniques.

[0064] Figure 40 A flowchart is shown for another example method for video processing based on currently disclosed techniques.

[0065] Figure 41 A flowchart is shown for another example method for video processing based on currently disclosed techniques.

[0066] Figure 42 A flowchart is shown for another example method for video processing based on currently disclosed techniques. Detailed Implementation

[0067] Due to the increasing demand for higher resolution video, video coding methods and technologies are ubiquitous in modern technology. Video codecs typically consist of electronic circuitry or software that compresses or decompresses digital video and are constantly being improved to provide higher coding efficiency. Video codecs convert uncompressed video into a compressed format and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end latency (delay). Compression formats typically conform to standard video compression specifications, such as the High Efficiency Video Coding (HEVC) standard (also known as H.265 or MPEG-H Part 2), the Universal Video Coding Standard to be Completed, or other current and / or future video coding standards.

[0068] Embodiments of the disclosed techniques can be applied to existing video coding standards (e.g., HEVC, H.265) and future standards to improve compression performance. Section headings are used in this document to improve readability of the description and do not in any way limit the discussion or embodiments (and / or implementations) to the relevant sections only.

[0069] 1. Example of inter-frame prediction in HEVC / H.265

[0070] Over the years, video coding standards have improved significantly and now offer, in part, high coding efficiency and support for higher resolutions. Latest standards such as HEVC and H.265 are based on a hybrid video coding architecture that utilizes temporal prediction plus transform coding.

[0071] 1.1. Examples of Predictive Models

[0072] Each inter-frame prediction PU (prediction unit) has motion parameters for one or two lists of reference images. In some embodiments, the motion parameters include motion vectors and reference image indices. In other embodiments, inter_pred_idc can also be used to signal the use of one of the two lists of reference images. In yet another embodiment, the motion vectors can be explicitly encoded as increments relative to the predictor.

[0073] When a CU is encoded using the skip mode, a PU is associated with a CU, and there are no significant residual coefficients, no encoded motion vector increments, or reference picture indices. A Merge mode is specified to obtain the motion parameters of the current PU from neighboring PUs, including spatial and temporal candidates. The Merge mode can be applied to any PU for inter-frame prediction, not just the skip mode. An alternative to the Merge mode is explicit transmission of motion parameters, where, for each PU, the motion vectors, the corresponding reference picture index for each reference picture list, and the reference picture list usage are explicitly communicated via signaling.

[0074] A PU is generated from a sample block when the signaling indicates that one of two lists of reference images will be used. This is called "uni-prediction". Uni-prediction can be used for both P-bands and B-bands.

[0075] When the signaling indicates that two lists of reference images will be used, PUs are generated from two sample blocks. This is called "bi-prediction". Bi-prediction applies only to B-bands.

[0076] 1.1.1.1 Constructing candidate implementations for the Merge pattern

[0077] When predicting the PU using the Merge pattern, the indices pointing to entries in the Merge candidate list are parsed from the bitstream and used to retrieve motion information. The construction of this list can be summarized in the following steps:

[0078] Step 1: Derivation of the original candidates

[0079] Step 1.1: Spatial Candidate Derivation

[0080] Step 1.2: Redundancy check of spatial candidates

[0081] Step 1.3: Derivation of Time Candidates

[0082] Step 2: Insert additional candidates

[0083] Step 2.1: Create bidirectional prediction candidates

[0084] Step 2.2: Insert zero-motion candidates

[0085] Figure 1 An example of constructing a Merge candidate list based on the sequence of steps summarized above is shown. For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five distinct positions. For temporal Merge candidate derivation, up to one Merge candidate is selected from two candidates. Since a constant number of candidates is assumed for each PU at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the stripe header. Because the number of candidates is constant, truncated unary binarization (TU) is used to encode the index of the best Merge candidate. If the CU size is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of a 2N×2N prediction unit.

[0086] 1.1.2 Construction Space Merge Candidates

[0087] In the derivation of the space Merge candidate, in the position of Figure 2Up to four merged candidates are selected from the candidates for the described positions. The derivation order is A1, B1, B0, A0, and B2. Position B2 is considered only if any PU at positions A1, B1, B0, or A0 is unavailable (e.g., because it belongs to another stripe or block) or if it is intra-coded. After adding the candidate at position A1, redundancy checks are performed on the remaining candidates to ensure that candidates with the same motion information are excluded from the list, thus improving coding efficiency. To reduce computational complexity, not all possible candidate pairs are considered in the aforementioned redundancy checks. Instead, only... Figure 3 Pairs connected by arrows are added to the list only if the corresponding candidate used for redundancy checking has different motion information. Another source of duplicate motion information is a "second PU" associated with partitions different from 2N×2N. As an example, Figure 4A and 4B The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In some embodiments, adding this candidate may result in two prediction units with the same motion information, which is redundant for a coding unit with only one PU. Similarly, when the current PU is partitioned into 2N×N, position B1 is not considered.

[0088] 1.1.1.3 Construction Time Merge Candidates

[0089] In this step, only one candidate is added to the list. Specifically, in the derivation of the merge candidate at this time, the scaled motion vector is derived based on the co-localized PU, which belongs to the image within a given list of reference images that has the smallest point of interest (POC) difference with the current image. The list of reference images to be used for deriving the co-localized PU is explicitly notified by signaling in the strip header.

[0090] Figure 5 An example of the derivation for scaling motion vectors for temporal merge candidates (e.g., dashed lines) is shown, which is scaled from the motion vectors of co-localized PUs using POC distances tb and td, where tb is defined as the POC difference between the current image and the reference image of the current image, and td is defined as the POC difference between the reference image of the co-localized image and the co-localized image. The reference image index for the temporal merge candidate is set to zero. For the B-strip, two motion vectors are obtained, one for reference image list 0 and the other for reference image list 1, and these two motion vectors are combined to obtain bidirectional predicted merge candidates.

[0091] In the common localization PU(Y) belonging to the reference frame, a temporal candidate position is selected between candidate C0 and C1, such as... Figure 6 As shown. If the PU at position C0 is unavailable, intra-coded, or outside the current CTU, then position C1 is used. Otherwise, position C0 is used for the derivation of the time merge candidate.

[0092] 1.1.4 Constructing Additional Types for Merge Candidates

[0093] In addition to the space-time merge candidates, two additional types of merge candidates exist: combined bidirectional predictive merge candidates and zero merge candidates. Combined bidirectional predictive merge candidates are generated by utilizing space-time merge candidates. These combined bidirectional predictive merge candidates are only used for B-strips. Combined bidirectional predictive candidates are generated by combining the motion parameters of the first reference image list of the original candidate with the motion parameters of the second reference image list of another candidate. If these two tuples provide different motion hypotheses, they will form a new bidirectional predictive candidate.

[0094] Figure 7 An example of this process is shown, where two candidates in the original list (710 on the left) have mvL0 and refIdxL0 or mvL1 and refIdxL1, which are used to create a bidirectional prediction of the Merge candidate combination added to the final list (on the right).

[0095] Zero-motion candidates are inserted to populate the remaining entries in the Merge candidate list, thus achieving the MaxNumMergeCand capacity. These candidates have null spatial displacements and reference image indices that start at zero and increase whenever a new zero-motion candidate is added to the list. The number of reference frames used for these candidates is 1 and 2, respectively, for unidirectional and bidirectional prediction. In some embodiments, redundancy checks are not performed on these candidates.

[0096] 1.1.5 Example of motion estimation region for parallel processing

[0097] To accelerate encoding processing, motion estimation can be performed in parallel, thereby simultaneously deriving the motion vectors of all prediction units within a given region. Deriving merge candidates from spatial neighborhoods can interfere with parallel processing because a prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is complete. To mitigate the trade-off between encoding efficiency and processing latency, a Motion Estimation Region (MER) can be defined. The size of the MER is signaled in the Picture Parameter Set (PPS) using the syntax element "log2_parallel_merge_level_minus2". When defining the MER, merge candidates falling into the same region are marked as unavailable and therefore not considered in the list construction.

[0098] 1.2 Examples of Advanced Motion Vector Prediction (AMVP)

[0099] AMVP utilizes the spatiotemporal correlation of motion vectors with adjacent PUs for explicit transmission of motion parameters. It constructs a motion vector candidate list by first verifying the availability above and to the left of temporally adjacent PU locations, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and send the corresponding index indicating the selected candidate. Similar to Merge index signaling, a truncated unary is used to encode the index of the best motion vector candidate. In this case, the maximum value to be encoded is 2 (see [link to documentation]). Figure 8 The following sections provide details of the derivation process for the motion vector prediction candidates.

[0100] 1.2.1 Example of constructing motion vector prediction candidates

[0101] Figure 8 The derivation process of motion vector prediction candidates is summarized, and it can be implemented by taking the index as input for each list of reference images.

[0102] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For the derivation of spatial motion vector candidates, the final result is based on the previously established... Figure 2 The motion vector derivation for each PU at the five different locations shown in the figure derives two motion vector candidates.

[0103] For temporal motion vector candidate derivation, a motion vector candidate is selected from two candidates, derived based on the locations of two distinct co-localizations. After generating the first list of spatiotemporal candidates, duplicate motion vector candidates are removed from the list. If the number of potential candidates is greater than two, motion vector candidates with a reference image index greater than 1 from the associated reference image list are removed from the list. If the number of spatiotemporal motion vector candidates is less than two, additional zero motion vector candidates are added to the list.

[0104] 1.2.2 Constructing Spatial Motion Vector Candidates

[0105] In the derivation of the spatial motion vector candidates, at most two candidates are considered from the five potential candidates, which are located as previously stated. Figure 2The derivation in the PU at the positions shown is the same as the position of the motion merge. The derivation order on the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order on the upper side of the current PU is defined as B0, B1, B2, and scaled B0, scaled B1, scaled B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require spatial scaling, and two of which do. The four different cases are summarized as follows:

[0106] No space scaling

[0107] -(1) Same list of reference images, and same reference image index (same POC) -(2) Different list of reference images, but the same reference images (same POC)

[0108] • Spatial scaling

[0109] -(3) Same list of reference images, but different reference images (different POCs)

[0110] -(4) Different lists of reference images, and different reference images (different POCs)

[0111] First, the case without spatial scaling is checked, then spatial scaling is checked. Spatial scaling is considered when the POC differs between the reference image of the adjacent PU and the reference image of the current PU, regardless of the reference image list. If all PUs of the left candidate are unavailable or intra-coded, scaling of the above motion vectors is allowed to aid in the parallel derivation of the left and top MV candidates. Otherwise, spatial scaling of the above motion vectors is not allowed.

[0112] like Figure 9 As shown in the example, for spatial scaling, the motion vectors of adjacent PUs are scaled in a similar manner to temporal scaling. One difference is that a list of reference images and the index of the current PU are given as input; the actual scaling process is the same as the temporal scaling process.

[0113] 1.2.3 Constructing Time Motion Vector Candidates

[0114] Except for the derivation of the reference image index, all the procedures used to derive the temporal Merge candidate are the same as those used to derive the spatial motion vector candidate (e.g., ...). Figure 6 (As shown in the example). In some embodiments, the reference image index is signaled to the decoder.

[0115] 2. Example of inter-frame prediction method in Joint Exploration Model (JEM)

[0116] In some embodiments, reference software called the Joint Exploration Model (JEM) is used to explore future video coding techniques. In the JEM, sub-block-based predictions are employed across several coding tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), bidirectional optical flow (BIO), frame rate upconversion (FRUC), local adaptive motion vector resolution (LAMVR), overlapping block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR).

[0117] 2.1 Example of motion vector prediction based on sub-CU

[0118] In a JEM with a quadtree plus binary tree (QTBT), each CU can have at most one set of motion parameters for each prediction direction. In some embodiments, two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The Alternative Temporal Motion Vector Prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the juxtaposed reference image. In the Spatial-Temporal Motion Vector Prediction (STMVP) method, the motion vectors of the sub-CUs are recursively derived using temporal motion vector prediction values ​​and spatially adjacent motion vectors. In some embodiments, motion compression of the reference frame may be disabled to preserve a more accurate motion field for sub-CU motion prediction.

[0119] 2.1.1 Example of Optional Time Motion Vector Prediction (ATMVP)

[0120] In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.

[0121] Figure 10 An example of the ATMVP motion prediction process for CU 1000 is shown. The ATMVP 1000 method predicts the motion vectors of sub-CUs 1001 within CU 1000 in two steps. The first step is to identify the corresponding block 1051 in reference image 1050 using time vectors. Reference image 1050 is also referred to as the motion source image. The second step is to divide the current CU 1000 into sub-CUs 1001 and obtain the motion vector and reference index of each sub-CU from the block corresponding to each sub-CU.

[0122] In the first step, reference image 1050 and corresponding blocks are determined based on the motion information of spatially adjacent blocks of the current CU 1000. To avoid repeated scanning of adjacent blocks, the first Merge candidate in the Merge candidate list of the current CU 1000 is used. The first available motion vector and its associated reference index are set to the index of the time vector and the motion source image. In this way, corresponding blocks can be identified more accurately compared to TMVP, where the corresponding block (sometimes called the juxtaposed block) is always located in the lower right or center position relative to the current CU.

[0123] In the second step, the corresponding block of sub-CU 1051 is identified by adding a time vector to the coordinates of the current CU, using the time vector in the motion source image 1050. For each sub-CU, the motion information of its corresponding block (e.g., the smallest motion grid covering the center sample) is used to derive the sub-CU's motion information. After identifying the motion information of the corresponding N×N blocks, it is converted into the reference index and motion vector of the current sub-CU in the same manner as the TMVP of HEVC, where motion scaling and other processes also apply. For example, the decoder checks whether a low-latency condition is met (e.g., the POC of all reference images of the current image is less than the POC of the current image) and may use the motion vector MV. x (For example, the motion vector corresponding to the reference image list X) to predict the motion vector MV of each sub-CU. y (For example, where X equals 0 or 1 and Y equals 1-X).

[0124] 2.1.2 Example of Space-Time Motion Vector Prediction (STMVP)

[0125] In the STMVP method, the motion vector of the sub-CU is recursively derived according to the raster scan order. Figure 11 An example of a CU with four sub-blocks and adjacent blocks is shown. Consider an 8×8 CU1100 containing four 4×4 sub-CUs A(1101), B(1102), C(1103), and D(1104). The adjacent 4×4 blocks in the current frame are labeled a(1111), b(1112), c(1113), and d(1114).

[0126] Motion derivation of sub-CU A begins by identifying its two spatial neighbors. The first neighbor is the N×N block above sub-CU A1101 (block c 1103). If block c (1113) is unavailable or intra-coded, the other N×N blocks above sub-CU A (1101) are checked (starting from block c 1113, from left to right). The second neighbor is the block to the left of sub-CU A1101 (block b 1112). If block b (1112) is unavailable or intra-coded, the other blocks to the left of sub-CU A1101 are checked (starting from block b 1112, from top to bottom). Motion information obtained from adjacent blocks in each list is scaled to the first reference frame of the given list. Next, the Temporal Motion Vector Predictor (TMVP) of sub-block A1101 is derived by following the same procedure as the TMVP derivation specified in HEVC. Motion information of the juxtaposed block at D 1104 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged separately for each reference list. The averaged motion vector is then designated as the motion vector for the current sub-CU.

[0127] 2.1.3 Example of motion prediction mode signaling for sub-CU

[0128] In some embodiments, a sub-CU mode is enabled as an additional Merge candidate, and no additional syntax elements are required to signal this mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent the ATMVP mode and the STMVP mode. In some embodiments, up to seven Merge candidates can be used if the sequence parameter set indicates that ATMVP and STMVP are enabled. The encoding logic of the additional Merge candidates is the same as that of the Merge candidates in HM, meaning that for each CU in a P or B stripe, the two additional Merge candidates may require two additional RD checks. In some embodiments, such as JEM, the binary (bin) of all Merge indices is context-encoded by CABAC (Context-Based Adaptive Binary Arithmetic Encoding). In other embodiments, such as HEVC, only the first binary is context-encoded, while the remaining binary is context-bypass encoded.

[0129] 2.2 Adaptive Motion Vector Differential Resolution

[0130] In some embodiments, when the use_integer_mv_flag in the stripe header is equal to 0, the motion vector difference (MVD) is signaled in units of quarter-luminance samples (between the motion vector of the PU and the predicted motion vector). In JEM, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, MVD can be encoded in units of quarter-luminance samples, integer luminance samples, or four luminance samples. MVD resolution is controlled at the coding unit (CU) level, and an MVD resolution flag is conditionally signaled for each CU having at least one non-zero MVD component.

[0131] For a CU with at least one non-zero MVD component, signaling informs a first flag to indicate whether quarter-luminance sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter-luminance sample MV precision is not used, signaling informs another flag to indicate whether integer luminance sample MV precision or four-luminance sample MV precision is used.

[0132] When the first MVD resolution flag of the CU is zero or not encoded for the CU (meaning all MVDs in the CU are zero), a quarter-luminance sample MV resolution is used for the CU. When the CU uses integer luminance sample MV precision or four-luminance sample MV precision, the MVPs in the CU's AMVP candidate list are rounded to the corresponding precision.

[0133] In the encoder, CU-level RD checks are used to determine which MVD resolution to use for the CU. That is, for each MVD resolution, CU-level RD checks are performed three times. To speed up the encoder, the following encoding scheme is applied in JEM.

[0134] • During the RD verification of a CU with a normal quarter-luminance sample MVD resolution, the motion information (integer luminance sample accuracy) of the current CU is stored. The stored motion information (after rounding) is used as the starting point for further small-range motion vector refinement for the same CU with integer luminance sample and 4-luminance sample MVD resolutions during the RD verification, so that the time-consuming motion estimation process is not repeated three times.

[0135] • Conditionally invoke the RD check for the CU with a 4-luminance sample MVD resolution. For the CU, skip the RD check for the 4-luminance sample MVD resolution if the RD cost integer luminance sample MVD resolution is much greater than a quarter luminance sample MVD resolution.

[0136] 2.3 Examples of Higher Motion Vector Storage Accuracy

[0137] In HEVC, motion vector accuracy is one-quarter of a pixel (one-quarter of the luminance sample and one-eighth of the chrominance sample from a 4:2:0 video). In JEM, the accuracy of internal motion vector storage and merge candidates is increased to 1 / 16 pixel. This higher motion vector accuracy (1 / 16 pixel) is used for motion-compensated inter-frame prediction of CUs encoded in skip / merge mode. For CUs encoded using normal AMVP mode, integer pixel or one-quarter pixel motion is used.

[0138] The SHVC upsampling interpolation filter, with the same filter length and normalization factor as the HEVC motion compensation interpolation filter, is used as the motion compensation interpolation filter for additional fractional pixel locations. The chroma component motion vector accuracy in JEM is 1 / 32 sample, and the additional interpolation filter for the 1 / 32 pixel fractional location is derived by averaging the filters for two adjacent 1 / 16 pixel fractional locations.

[0139] 2.4 Example of Overlapped Block Motion Compensation (OBMC)

[0140] In JEM, OBMC can be enabled and disabled using CU-level syntax. When using OBMC in JEM, OBMC is performed on all Motion Compensation (MC) block boundaries except for the right and bottom boundaries of the CU. Furthermore, it is applied to both luma and chroma components. In JEM, MC blocks correspond to coded blocks. When encoding a CU using sub-CU modes (including sub-CUE, affine, and FRUC modes), each sub-block of the CU is an MC block. To handle CU boundaries uniformly, OBMC is performed at the sub-block level for all MC block boundaries, where the sub-block size is set to equal to 4×4, such as... Figure 12A and 12B As shown.

[0141] Figure 12A The sub-block at the CU / PU boundary is shown; the shaded sub-block is where the OBMC is applied. Similarly, Figure 12B The sub-blocks in the ATMVP pattern are shown.

[0142] When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of the four connected adjacent sub-blocks (if available and different from the current motion vector) are also used to derive the prediction block for the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal for the current sub-block.

[0143] The prediction block based on the motion vectors of neighboring sub-blocks is represented as P.N , where N indicates the indices of the adjacent upper, lower, left, and right sub-blocks, and the predicted block based on the motion vector of the current sub-block is represented as P. C When P N When the motion information is based on the motion information of adjacent sub-blocks that contain the same motion information as the current sub-block, it does not start from P. N Execute OBMC. Otherwise, execute each P. N Sample added to P C In the same samples, that is, P N Add four rows / columns to P C The weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P. N And the weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are applied to the PC. An exception is small MC blocks (i.e., when the height or width of the coded block is equal to 4 or the CU is encoded using a sub-CU mode), for which only the P is applied. N Add two rows / columns to P C In this case, the weighting factors {1 / 4, 1 / 8} are used for P. N And the weighting factors {3 / 4, 7 / 8} are used for P. C For P generated based on the motion vectors of vertically (horizontally) adjacent sub-blocks N , will P N Samples in the same row (column) are added to P with the same weighting factor. C .

[0144] In JEM, for CUs with a size of 256 lumen samples or less, signaling informs a CU-level flag to indicate whether OBMC should be applied to the current CU. For CUs with a size exceeding 256 lumen samples or not encoded using AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to the CU, its effects are considered during the motion estimation phase. The predicted signal formed by OBMC using motion information from the upper and left adjacent blocks is used to compensate for the upper and left boundaries of the original signal of the current CU, and then normal motion estimation processing is applied.

[0145] 2.5 Example of Local Illumination Compensation (LIC)

[0146] Illumination compensation (LIC) is based on a linear model for illumination variations, using a scaling factor a and an offset b. It is adaptively enabled or disabled for each inter-frame mode coding unit (CU).

[0147] When LIC is applied to CU, the least squares error method is used to derive parameters a and b by using the neighboring samples of the current CU and their corresponding reference samples. Figure 13An example of neighboring samples used to derive the parameters of the IC algorithm is shown. More specifically, as... Figure 13 As shown, neighboring samples from the subsampled CU (2:1 subsample) and corresponding samples from the reference image (identified by the motion information of the current CU or subCU) are used. The IC parameters are derived and applied to each prediction direction.

[0148] When encoding a CU in Merge mode, the LIC flag is copied from the adjacent block in a manner similar to motion information copying in Merge mode; otherwise, the CU signaling notifies the LIC flag to indicate whether LIC should be applied.

[0149] When LIC is enabled for an image, additional CU-level RD checks are required to determine whether LIC should be applied to the CU. When LIC is enabled for the CU, Mean-Removed Sum Of Absolute Difference (MR-SAD) and Mean-Removed Sum Of Absolute Hadamard-Transformed Difference (MR-SATD) are used instead of SAD and SATD for integer pixel motion search and fractional pixel motion search, respectively.

[0150] To reduce coding complexity, the following coding scheme is applied in JEM.

[0151] When there is no significant lighting change between the current image and its reference images, LIC is disabled for the entire image. To identify this situation, histograms of the current image and each reference image of the current image are calculated at the encoder. If the histogram difference between the current image and each reference image of the current image is less than a given threshold, LIC is disabled for the current image; otherwise, LIC is enabled for the current image.

[0152] 2.6 Examples of Affine Motion Compensation Prediction

[0153] In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). However, the camera and object may exhibit various motions, such as zooming in / out, rotation, perspective motion, and / or other irregular motions. On the other hand, JEM applies a simplified affine transformation motion compensation prediction. Figure 14 An example of the affine motion field of block 1400 described by two control point motion vectors V0 and V1 is shown. The motion vector field (MVF) of block 1400 is described by the following equation:

[0154]

[0155] like Figure 14 As shown, (v 0x ,v 0y (v) is the motion vector of the top-left control point. 1x ,v 1y ) is the motion vector of the upper right control point.

[0156] To further simplify motion compensation prediction, sub-block-based affine transformation prediction can be applied. The sub-block size M×N is derived as follows:

[0157]

[0158] Here, MvPre is the fractional accuracy of the motion vector (e.g., 1 / 16 in JEM), (v 2x ,v 2y ) is the motion vector of the lower left control point calculated according to Equation 1. If necessary, M and N can be adjusted downwards to make them the divisors of w and h, respectively.

[0159] Figure 15 An example of the affine MVF for each sub-block of block 1500 is shown. To derive the motion vector for each M×N sub-block, the motion vector of the center sample of each sub-block is calculated according to Equation 1 and rounded to the motion vector fractional accuracy (e.g., 1 / 16 in JEM). A motion-compensated interpolation filter is then applied to generate a prediction for each sub-block using the derived motion vector.

[0160] After MCP, the high-accuracy motion vector of each sub-block is rounded and saved with the same accuracy as the normal motion vector.

[0161] In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. AF_INTER mode can be applied to CUs with both width and height greater than 8. The affine flag at the CU level is signaled in the bitstream to indicate whether AF_INTER mode is used. In AF_INTER mode, adjacent blocks are used to construct motion vector pairs {(v0,v1)|v0={v...}. A ,v B ,v c},v1={v D ,v E The candidate list of}}.

[0162] Figure 16 An example of motion vector prediction (MVP) for block 1600 in AF_INTER mode is shown. Figure 16As shown, v0 is selected from the motion vectors of sub-blocks A, B, or C. Motion vectors from neighboring blocks can be scaled according to a reference list. The scaling can also be based on the relationship between the Picture Order Count (POC) for the reference of the neighboring block, the POC for the reference of the current CU, and the POC of the current CU. The method for selecting v1 from neighboring sub-blocks D and E is similar. If the number of candidates in the candidate list is less than two, the list can be populated by motion vector pairs formed by copying each AMVP candidate. When the candidate list is greater than two, the candidates can first be sorted according to adjacent motion vectors (e.g., based on the similarity of two motion vectors in a candidate pair). In some embodiments, the first two candidates are retained. In some embodiments, a rate distortion (RD) cost check is used to determine which motion vector pair candidate is selected as the Control Point Motion Vector Prediction (CPMVP) for the current CU. The index indicating the position of the CPMVP in the candidate list can be signaled in the bitstream. After determining the CPMVP of the current affine CU, affine motion estimation is applied and the Control Point Motion Vector (CPMV) is found. Then, the difference between CPMV and CPMVP is communicated in the bitstream using signaling.

[0163] When CU is applied in AF_MERGE mode, it obtains the first block encoded in affine mode from the valid adjacent reconstructed blocks. Figure 17A This shows an example of the current candidate block selection order for the CU 1700. (Example:) Figure 17A As shown, the selection order can be from the left (1701), top (1702), top right (1703), bottom left (1704) to top left (1705) of the current CU 1700. Figure 17B Another example of a candidate block for the current CU 1700 in AF_MERGE mode is shown. If the adjacent lower-left block 1701 is encoded in affine mode, such as... Figure 17B As shown, the motion vectors v2, v3, and v4 of the upper left, upper right, and lower left corners of the CU containing block A are derived. Furthermore, the motion vector v0 of the upper left corner of the current CU 1700 is calculated based on v2, v3, and v4. The motion vector v1 of the upper right corner of the current CU can be calculated accordingly.

[0164] After calculating the CPMV v0 and v1 of the current CU according to the affine motion model in equation (1), the MVF of the current CU can be generated. In order to identify whether the current CU is encoded in AF_MERGE mode, when at least one adjacent block is encoded in affine mode, the affine flag can be notified by signaling in the bit stream.

[0165] 2.7 Example of Motion Vector Derivation (PMMVD) for Pattern Matching

[0166] PMMVD mode is a special merge mode based on the Frame-Rate Up Conversion (FRUC) method. This mode uses the decoder to derive block motion information instead of sending signaling to notify the block of its motion.

[0167] When the Merge flag of the CU is true, the FRUC flag can be signaled to the CU. When the FRUC flag is false, the Merge index can be signaled and the regular Merge mode can be used. When the FRUC flag is true, additional FRUC mode flags can be signaled to indicate which method (e.g., bilateral matching or template matching) will be used to derive the block's motion information.

[0168] On the encoder side, the decision on whether to use the FRUC Merge mode for the CU is based on the RD cost selection made for normal Merge candidates. For example, multiple matching modes for the CU (e.g., bilateral matching and template matching) are validated using RD cost selection. The matching mode that results in the minimum cost is further compared with other CU modes. If the FRUC matching mode is the most efficient mode, the FRUC flag is set to true for the CU, and the relevant matching mode is used.

[0169] Typically, the motion derivation process in the FRUC Merge model has two steps: first, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, the original motion vector of the entire CU is derived based on bilateral matching or template matching. First, a candidate MV list is generated, and the candidate that causes the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed near the starting point. The MV result with the minimum matching cost is taken as the MV of the entire CU. Subsequently, using the derived CU motion vector as the starting point, the motion information is further refined at the sub-CU level.

[0170] For example, the following derivation process is performed for the motion information derivation of a W×HCU. In the first stage, the MV of the entire W×HCU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as shown in (3), and D is a predefined partitioning depth, which is set to 3 by default in JEM. Then the MV of each sub-CU is derived.

[0171]

[0172] Figure 18An example of bilateral matching used in the Frame Rate Upconversion (FRUC) method is shown. Bilateral matching is used to derive motion information of the current CU (1800) by finding the closest match between two blocks along the motion trajectory of the current CU (1800) in two different reference images (1810, 1811). Under the assumption of continuous motion trajectories, the motion vectors MV0 (1801) and MV1 (1802) pointing to the two reference blocks are proportional to the temporal distances between the current image and the two reference images (e.g., TD0 (1803) and TD1 (1804)). In some embodiments, bilateral matching becomes a mirror-based bidirectional MV when the current image 1800 is temporally between the two reference images (1810, 1811) and the temporal distances from the current image to the two reference images are the same.

[0173] Figure 19 An example of template matching used in the Frame Rate Upconversion (FRUC) method is shown. Template matching is used to deduce motion information for the current CU 1900 by finding the closest match between a template in the current image 1910 (the top and / or left adjacent block of the current CU) and a block in the reference image (e.g., of the same size as the template). In addition to the FRUCMerge mode described above, template matching can also be applied to the AMVP mode. As done in both JEM and HEVC, AMVP has two candidates. New candidates are deduced using the template matching method. If a newly deduced candidate by template matching differs from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and the list size is then set to 2 (e.g., by removing the second existing AMVP candidate). When applied to AMVP mode, only CU-level search is performed.

[0174] The MV candidates set at the CU level may include the following: (1) the original AMVP candidate if the current CU is in AMVP mode, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and the top and left adjacent motion vectors.

[0175] When using bilateral matching, each valid MV of the Merge candidate can be used as input to generate MV pairs assuming bilateral matching. For example, in reference list A, a valid MV of the Merge candidate is (MVa, refa). Then, in another reference list B, a reference image refb for its paired bilateral MV is found such that refa and refb are on different sides of the current image in time. If such a refb is not available in reference list B, then refb is determined to be a different reference from refa, and its temporal distance to the current image is the minimum in list B. After determining refb, MVb is derived by scaling MVa based on the temporal distance between refa and refb in the current image.

[0176] In some embodiments, four MVs from the interpolated MV field can also be added to the CU-level candidate list. More specifically, the interpolated MVs at the current CU positions (0,0), (W / 2,0), (0,H / 2), and (W / 2,H / 2) are added. When FRUC is applied to the AMVP pattern, the original AMVP candidates are also added to the CU-level MV candidate set. In some embodiments, at the CU level, 15 MVs are added to the candidate list for AMVPCU and 13 MVs are added to the candidate list for MergeCU.

[0177] The MV candidates set at the sub-CU level include: (1) MVs determined from the CU level search, (2) adjacent MVs at the top, left, top-left, and top-right corners, (3) scaled versions of the juxtaposed MVs from the reference image, (4) one or more ATMVP candidates (maximum 4), and (5) one or more STMVP candidates (e.g., maximum 4). The scaled MVs from the reference image are derived as follows. The reference images in both lists are traversed. The MVs at the juxtaposed positions of the sub-CUs in the reference images are scaled to the reference of the starting CU level MV. The ATMVP and STMVP candidates may be limited to the first four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

[0178] Generation of interpolated MV fields. Before encoding the frames, an interpolated motion field is generated for the entire image based on a one-sided ME. The motion field can then be used later as a CU-level or sub-CU-level MV candidate.

[0179] In some embodiments, the motion field of each reference image in the two reference lists is traversed in a 4×4 block-level manner. Figure 20An example of one-sided motion estimation (ME)2000 in the FRUC method is shown. For each 4×4 block, if the motion associated with the block passes through 4×4 blocks in the current image and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current image according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the block's motion is marked as unavailable in the interpolated motion field.

[0180] Interpolation and matching costs. When the motion vector points to the location of a fractional sample, motion-compensated interpolation is required. To reduce complexity, bilinear interpolation can be used for bilateral matching and template matching instead of the conventional 8-tap HEVC interpolation.

[0181] The calculation of matching cost differs slightly at different steps. When selecting candidates from the candidate set at the CU level, the matching cost can be the absolute sum difference (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C of bilateral matching in the sub-CU level search is calculated as follows:

[0182]

[0183] Here, w is the weighting factor. In some embodiments, w can be set to 4. MV and MV s These indicate the current MV and the starting MV, respectively. SAD can still be used as the matching cost for template matching in sub-CU level searches.

[0184] In FRUC mode, the motion signature (MV) is derived using only luma samples. The derived motion is used for both luma and chroma predictions in the inter-frame prediction (MC). After determining the MV, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

[0185] MV refinement is a pattern-based MV search, categorized by bilateral matching cost or template matching cost. JEM supports two search modes—Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search—for MV refinement at the CU and sub-CU levels, respectively. For both CU and sub-CU level MV refinement, MV is directly searched with MV precision based on one-quarter of the luminance samples, followed by MV refinement based on one-eighth of the luminance samples. The search range for MV refinement in the CU and sub-CU steps is set to equal 8 luminance samples.

[0186] In the bilateral matching Merge mode, bidirectional prediction is applied because the motion information of the CU is derived based on the nearest match between two blocks along the current CU's motion trajectory in two different reference images. In the template matching Merge mode, the encoder can select for the CU from unidirectional predictions in list 0, unidirectional predictions in list 1, or bidirectional predictions. The template matching cost can be selected based on the following:

[0187] If costBi <= factor * min(cost0, cost1)

[0188] Then use bidirectional prediction;

[0189] Otherwise, if cost0 <= cost1

[0190] Then use the one-way prediction from list 0;

[0191] otherwise,

[0192] Use the one-way prediction from List 1;

[0193] Where cost0 is the SAD of template matching in list 0, cost1 is the SAD of template matching in list 1, and costBi is the SAD of bidirectional prediction template matching. For example, when the value of factor equals 1.25, this means that the selection process is biased towards bidirectional prediction. Inter-frame prediction direction selection can be applied to CU-level template matching processes.

[0194] 2.8 Examples of Bidirectional Optical Flow (BIO)

[0195] In BIO, motion compensation is first performed to generate a first prediction for the current block (in each prediction direction). This first prediction is used to derive the spatial gradient, temporal gradient, and optical flow for each sub-block / pixel within the block, and is then used to generate a second prediction, such as the final prediction for the sub-block / pixel. Details are described below.

[0196] Bidirectional optical flow (BIO) is a sample-level motion refinement that is performed on top of block-by-block motion compensation for bidirectional prediction. In some embodiments, sample-level motion refinement does not use signaling.

[0197] Let I (k) The brightness value of reference k (k=0,1) after block motion compensation, and I (k) The horizontal and vertical components of the gradient. Assuming optical flow is effective, the motion vector field (v) x ,v y The following formula is given:

[0198]

[0199] Combining this optical flow equation with the Hermitian interpolation of each sample's motion trajectory yields a unique third-order polynomial, which ultimately matches the function value I. (k) and its derivative Both. The value of the third-order polynomial at t=0 is the BIO prediction:

[0200]

[0201] Figure 21 An example optical flow trajectory in the bidirectional optical flow (BIO) method is shown. Here, τ0 and τ1 represent the distances to the reference frame, such as... Figure 21 As shown. Distances τ0 and τ1 are calculated based on the POC of Ref0 and Ref1: τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). If both predictions originate from the same time direction (both from the past or both from the future), then the sign is different, i.e., τ0·τ1 < 0. In this case, BIO is applied only when the predictions do not originate from the same time (i.e., τ0 ≠ τ1), both reference regions have non-zero motion (MVx0, MVy0, MVx1, MVy1 ≠ 0), and the block motion vector is proportional to the time distance (MVx0 / MVx1 = MVy0 / MVy1 = -τ0 / τ1).

[0202] The motion vector field (v) is determined by minimizing the difference Δ between the values ​​at points A and B. x ,v y First linear term:

[0203]

[0204] All values ​​in the above equation depend on the sample location, denoted as (i′, j′). Assuming the motion is consistent in the local surrounding region, minimize Δ within a square window Ω of (2M+1)×(2M+1) centered on the current prediction point, where M equals 2:

[0205]

[0206] For this optimization problem, JEM uses a simplification method, first minimizing in the vertical direction and then minimizing in the horizontal direction. This results in the following:

[0207]

[0208]

[0209] in,

[0210]

[0211] To avoid division by zero or very small values, regularization parameters r and m can be introduced in equations 9 and 10.

[0212] r = 500·4 d-8 (12)

[0213] m = 700·4 d-8 (13)

[0214] Here, d is the bit depth of the video sample.

[0215] To ensure that BIO memory access remains the same as regular bidirectional predictive motion compensation, all predicted and gradient values ​​I are computed only for the position within the current block. (k) , Figure 22A An example of an access location outside block 2200 is shown. Figure 22A As shown, in equation (9), a (2M+1)×(2M+1) square window Ω centered on the current prediction point on the boundary of the prediction block needs to access locations outside the block. In JEM, the I outside the block... (k) , The value is set to be equal to the nearest available value within the block. For example, this could be implemented to fill region 2201, such as... Figure 22B As shown.

[0216] Using BIO, the motion field can be refined for each sample. To reduce computational complexity, a block-based BIO design is used in JEM. Motion refinement can be calculated based on 4×4 blocks. In block-based BIO, the s in Equation 9 of all samples in a 4×4 block can be aggregated. n The value of s, then s n The aggregated values ​​are used to derive the BIO motion vector offset for a 4×4 block. More specifically, the following formula can be used for block-based BIO derivation:

[0217]

[0218] Here b k Let represent the sample set belonging to the k-th 4×4 block of the prediction block. Then, in equations 9 and 10, s... n Replace with ((s) n,bk )>>4), to derive the associated motion vector offset.

[0219] In some scenarios, the MV regiment in BIO may be unreliable due to noise or irregular motion. Therefore, in BIO, the size of the MV regiment is thresholded. The threshold is determined based on whether all reference images of the current image come from the same direction. For example, if all reference images of the current image come from the same direction, the threshold value is set to 12×2. 14-d Otherwise, set it to 12×2 13-d .

[0220] The gradient of the BIO can be simultaneously computed using motion-compensated interpolation that operates in accordance with the HEVC motion compensation process (e.g., 2D separable finite impulse response (FIR)). In some embodiments, the input to the 2D separable FIR is a reference frame sample that is identical to the motion compensation process and the fractional position (fracX, fracY) based on the fractional portion of the block motion vector. For the horizontal gradient... First, the vertically interpolated signal is used using BIOfilterS, corresponding to the fractional position fracY with a descaling offset of d-8. Then, a gradient filter BIOfilterG is applied in the horizontal direction, corresponding to the fractional position fracX with a descaling offset of 18-d. For the vertical gradient... First, a gradient filter is applied vertically using BIOfilterG, corresponding to the fractional position fracY with a descaling offset of d-8. Then, signal displacement is performed horizontally using BIOfilterS, corresponding to the fractional position fracX with a descaling offset of 18-d. The lengths of the interpolation filter BIOfilterG for gradient calculation and the interpolation filter BIOfilterS for signal displacement can be relatively short (e.g., 6 taps) to maintain reasonable complexity. Table 1 shows examples of filters that can be used for gradient calculation at different fractional positions of block motion vectors in BIO. Table 2 shows examples of interpolation filters that can be used for predictive signal generation in BIO.

[0221] Table 1: Exemplary filters for gradient computation in BIO

[0222] Fractional pixel position Gradient interpolation filter (BIOfilterG) 0 {8,-39,-3,46,-17,5} 1 / 16 {8,-32,-13,50,-18,5} 1 / 8 {7,-27,-20,54,-19,5} 3 / 16 {6,-21,-29,57,-18,5} 1 / 4 {4,-17,-36,60,-15,4} 5 / 16 {3,-9,-44,61,-15,4} 3 / 8 {1,-4,-48,61,-13,3} 7 / 16 {0,1,-54,60,-9,2} 1 / 2 {-1,4,-57,57,-4,1}

[0223] Table 2: Exemplary interpolation filters used for predictive signal generation in BIO

[0224] Fractional pixel position Interpolation filters for predicted signals (BIOfilters) 0 {0,0,64,0,0,0} 1 / 16 {1,-3,64,4,-2,0} 1 / 8 {1,-6,62,9,-3,1} 3 / 16 {2,-8,60,14,-5,1} 1 / 4 {2,-9,57,19,-7,2} 5 / 16 {3,-10,53,24,-8,2} 3 / 8 {3,-11,50,29,-9,2} 7 / 16 {3,-11,44,35,-10,3} 1 / 2 {3,-10,35,44,-11,3}

[0225] In JEM, BIO can be applied to all dual-prediction blocks when the two predictions come from different reference images. BIO can be disabled when Local Illumination Compensation (LIC) is enabled for a CU.

[0226] In some embodiments, OBMC is applied to blocks after the normal MC process. To reduce computational complexity, BIO may not be applied during the OBMC process. This means that BIO is only applied to the block's MC process when its own MV is used, and not during the OBMC process when the MV of an adjacent block is used.

[0227] 2.9 Example of Decoder-Side Motion Vector Refinement (DMVR)

[0228] In bidirectional prediction, for the prediction of a block region, two prediction blocks formed by the motion vectors (MV) of list0 and list1 are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined through a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral templates and reconstructed samples in the reference image to obtain a refined MV without transmitting additional motion information.

[0229] In DMVR, the bilateral template is generated as a weighted combination (i.e., average) of the two prediction blocks from the original MV0 of list 0 and the MV1 of list 1, respectively, as follows: Figure 23 As shown. The template matching operation involves calculating a cost metric between the generated template and the sample regions (around the original predicted block) in the reference images. For each of the two reference images, the MV that produces the minimum template cost is considered as the updated MV of that list to replace the original MV. In JEM, nine MV candidates are searched for each list. These nine MV candidates include the original MV and eight surrounding MVs that have a brightness sample offset from the original MV in the horizontal or vertical direction or both directions. Finally, the two new MVs, i.e., as shown... Figure 23 MV0′ and MV1′ shown are used to generate the final bidirectional prediction results. The sum of absolute differences (SAD) is used as the cost metric.

[0230] DMVR is applied to the Merge pattern of bidirectional prediction, where one MV comes from a past reference image and the other from a future reference image, without transferring additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or subCU Merge candidates are enabled for a CU.

[0231] 3CABAC Modification Examples

[0232] In JEM, compared to the design in HEVC, CABAC includes the following three main changes:

[0233] Context modeling for modification of transform coefficients;

[0234] Multi-hypothesis probability estimation with context-dependent update speed;

[0235] Adaptive primitiveization for contextual models.

[0236] 3.1 Example of contextual modeling for transform coefficients

[0237] In HEVC, transform coefficients of a coded block are encoded using non-overlapping coefficient groups (CGs), with each CG containing the coefficients of a 4×4 block of the coded block. CGs within a coded block and transform coefficients within a CG are encoded according to a predefined scan order. The encoding of a transform coefficient level of a CG with at least one non-zero transform coefficient can be divided into multiple scan channels. In the first channel, the first binary symbol (denoted by bin0, also known as significant_coeff_flag, which indicates that the size of the coefficient is greater than 0) is encoded. Next, two scan channels (denoted by bin1 and bin2, also known as coeff_abs_greater1_flag and coeff_abs_greater2_flag, respectively) can be applied for context encoding of the second / third binary symbols (bins). Finally, if necessary, more than two scan channels are invoked for encoding symbol information, along with the remaining values ​​of the coefficient level (also known as coeff_abs_level_remaining). Only binary symbols in the first three scan channels are encoded in regular mode, and these binary symbols are referred to as regular binary symbols in the following description.

[0238] In JEM, the context modeling of regular binary symbols is modified. When binary symbol i is encoded in the i-th scan channel (i = 0, 1, 2), the context index depends on the value of the i-th binary symbol in the neighborhood of previously encoded coefficients covered by the local template. Specifically, the context index is determined based on the sum of the i-th binary symbols of neighboring coefficients.

[0239] like Figure 24 As shown, a local template contains up to five spatially adjacent transform coefficients, where x represents the position of the current transform coefficient, and xi (i is from 0 to 4) indicates its five neighbors. To capture the characteristics of transform coefficients at different frequencies, a coding block can be divided into up to three regions, and the division method is fixed regardless of the coding block size. For example, when encoding bin0 of the luminance transform coefficient, as... Figure 24As shown, a coded block is divided into three regions marked with different colors, and a context index is listed for each region. Luminance and chrominance components are handled in a similar manner, but with separate sets of context models. Furthermore, the selection of the context model for bin0 (e.g., the valid marker) of the luminance component also depends on the transform size.

[0240] 3.2 Examples of Multiple Hypothesis Probability Estimation

[0241] The binary arithmetic encoder applies a "multi-hypothesis" probability update model based on two probability estimates P0 and P1 associated with each context model, and updates them independently at different adaptive rates as follows:

[0242]

[0243] in and Let M represent the probabilities before and after decoding the binary bit, respectively. i (4, 5, 6, 7) are parameters that control the probability update speed of the context model with index i; and k represents the precision of the probability (here equals 15).

[0244] The probability estimate P for interval subdivision in a binary arithmetic encoder is the mean of the estimates from two hypotheses:

[0245] P = (P0) new +P1 new ) / 2 (16)

[0246] In JEM, the parameter M used in equation (15) is assigned as follows to control the probability update rate of each context model. i The value of .

[0247] On the encoder side, the encoded binary bits associated with each context model are recorded. After encoding a stripe, for each context model with index i, the encoding is computed using different M bits. i Rate costs are calculated for values ​​(4, 5, 6, 7), and the one providing the lowest rate cost is selected. For simplicity, this selection process is only performed when a new combination of strip type and strip-level quantization parameters is encountered.

[0248] For each context model i, a 1-bit flag is sent to indicate M. i Is it different from the default value of 4? When the flag is 1, two bits are used to indicate M. i Is it equal to 5, 6, or 7?

[0249] 3.3 Initialization of the Context Model

[0250] Instead of using a fixed table for context model initialization in HEVC, the initial probability states of the context models for inter-frame coding slices can be initialized by copying states from previously encoded images. More specifically, after encoding the center-localized CTU for each image, the probability states of all context models are stored to serve as the initial states for the corresponding context models on subsequent images. In JEM, the initial state set for each inter-frame coding slice is copied from the stored states of previously encoded images with the same slice type and the same slice level QP as the current slice. This lacks loss robustness but is used for coding efficiency experiments in the current JEM scheme.

[0251] 4 Examples of related embodiments and methods

[0252] Methods related to the disclosed technology include extended LAMVR where the supported motion vector resolution ranges from 1 / 4 pixel to 4 pixels (1 / 4 pixel, 1 / 2 pixel, 1-pixel, 2-pixel, and 4-pixel). Information regarding the motion vector resolution is notified at the CU level via signaling when MVD information is communicated.

[0253] Depending on the CU's resolution, the CU's motion vectors (MV) and motion vector prediction (MVP) are adjusted. If the applied motion vector resolution is represented as R (R can be 1 / 4, 1 / 2, 1, 2, 4), then MV(MV x MV y ) and MVP (MVP) x MVP y The following is represented:

[0254] (MV x ,MV y ) = (Round(MV x / (R*4))*(R*4),Round(MV y / (R*4))*(R*4)) (17)

[0255] (MVP x MVP y ) = (Round(MVP) x / (R*4))*(R*4),Round(MVP y / (R*4))*(R*4)) (18)

[0256] Since both motion vector prediction and MV are adjusted through adaptive resolution, MVD (MVD x MVD y It is also aligned with the resolution, and is notified via signaling as follows based on the resolution:

[0257] (MVD xMVD y )=((MV x –MVP x ) / (R*4),(MV y –MVP y ) / R*4)) (19)

[0258] In this proposal, the Motion Vector Resolution Index (MVR Index) indicates both the MVP Index and the motion vector resolution. As a result, the proposed method does not have MVP Index signaling. The table below shows what each value of the MVR Index represents.

[0259] Table 3: Examples of MVR Index Representations

[0260] MVR Index 0 1 2 3 4 Pixel resolution (R) 1 / 4 1 / 2 1 2 4 The Nth MVP <![CDATA[1 st MVP]]> <![CDATA[2 nd MVP]]> <![CDATA[3 rd MVP]]> <![CDATA[4 th MVP]]> <![CDATA[5 th MVP]]>

[0261] In the case of bidirectional prediction, AMVR has three modes for each resolution. The AMVR bidirectional index (Bi-Index) indicates whether signaling is sent to notify the MVDx and MVDy of each reference list (list 0 or list 1). An example definition of the AMVR bidirectional index is shown in the table below.

[0262] Table 4: Examples of AMVP bidirectional indexes

[0263] AMVR bidirectional index <![CDATA[(MVD of List 0 x , MVD y )]]> <![CDATA[(MVD of List 1 x , MVD y )]]> 0 Signaling notification Signaling notification 1 No signaling notification Signaling notification 2 Signaling notification No signaling notification

[0264] 5. Disadvantages of existing implementation methods

[0265] In a current implementation using BIO, the MV calculated between the reference block / sub-block in list 0 (represented by refblk0) and the reference block / sub-block in list 1 (refblk1) is determined by (v x ,v y This indicates that it is only used for motion compensation of the current block / sub-block, and not for motion prediction, deblocking, OBMC, etc. in future coded blocks, which may be inefficient. For example, (v) can be generated for each sub-block / pixel of the block. x ,v y ), and formula (7) can be used to generate a second prediction for sub-blocks / pixels. However, (v x ,v y It is not used for motion compensation of sub-blocks / pixels, which may also be inefficient.

[0266] In another existing implementation that uses DMVR and BIO for bidirectional prediction of the PU, first, DMVR is performed. Then, the motion information of the PU is updated. Finally, BIO is performed using the updated motion information. That is, the input to BIO depends on the output of DMVR.

[0267] In another existing implementation using OBMC, for AMVP mode, for small blocks (width * height <= 256), the encoder determines whether to enable OBMC and notifies the decoder via signaling. This increases the complexity of the encoder. Furthermore, for a given block / sub-block, when OBMC is enabled, it always applies to both luma and chroma, which can lead to decreased encoding efficiency.

[0268] In another existing implementation using the AF_INTER mode, MVD needs to be encoded; however, it can only be encoded with 1 / 4 pixel precision, which may be inefficient.

[0269] 6. Example method for two-step inter-frame prediction for visual media coding

[0270] The embodiments of the currently disclosed technology overcome the shortcomings of existing implementations and provide additional solutions, thereby providing video coding with higher coding efficiency. Based on the disclosed technology, two-step inter-frame prediction can enhance existing and future video coding standards, as illustrated in the examples described below for various implementations. The examples of the disclosed technology provided below illustrate general concepts and are not intended to be construed as limiting. In the examples, the various features described in these examples can be combined unless explicitly indicated otherwise.

[0271] Regarding terminology, the reference images for the current images from List 0 and List 1 are denoted as Ref0 and Ref1, respectively. This represents τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current), and the reference blocks for the current blocks from Ref0 and Ref1 are denoted as refblk0 and refblk1, respectively. For a sub-block within the current block, the original MV of the corresponding sub-block in refblk0 pointing to refblk1 is represented as (v x ,v y The MV of the sub-blocks in Ref0 and Ref1 are respectively determined by (mvL0). x ,mvL0 y ) and (mvL1 x mvL1 y ) represents. (v) x ,v y ) represents the derived MV from the original MV in BIO. As described in this patent document, the updated motion vector-based method for motion prediction can be extended to existing and future video coding standards.

[0272] Example 1.MV(v x ,v y ) and MV (mvLX) x ,mvLX y), where X = 0 or 1, should be scaled to the same precision before the addition operation, such as before performing the techniques in Example 1(e) and / or Example 2.

[0273] (a) In one example, the target precision (to be scaled to) is set to the higher (for better performance) / lower (for lower complexity) precision between MV(v x ,v y ) and MV(mvLX x ,mvLX y ). Alternatively, regardless of the precision of these two MVs, the target precision (to be scaled to) is set to a fixed value (e.g., 1 / 32 pixel precision).

[0274] (b) In one example, the original MV (mvLX x ,mvLX y ) can be scaled to a higher precision before the addition operation. For example, it can be scaled from 1 / 4 pixel precision to 1 / 16 pixel precision. In this case, mvLX x = sign(mvLX x ) * (abs(mvLX x ) << N), mvLX y = sign(mvLX y ) * (abs(mvLX y ) << N), where the function sign(·) returns the sign of the input parameter (as shown below), and the function abs(·) returns the absolute value of the input parameter, and N = log2(curr_mv_precision / targ_mv_precision), and curr_mv_precision and targ_mv_precision are the current MV precision and the target MV precision respectively. For example, if the MV is scaled from 1 / 4 pixel precision to 1 / 16 pixel precision, then N = log2((1 / 4) / (1 / 16)) = 2.

[0275]

[0276] (i) Alternatively, mvLX x = mvLX x << N, mvLX y = mvLX y << N.

[0277] (ii) Alternatively, mvLX x = mvLX x << (N + K), mvLX y = mvLX y << (N + K).

[0278] (iii) Alternatively, mvLX x =sign(mvLX) x )*(abs(mvLX x )<<(N+K)),mvLX y =sign(mvLX) y )*(abs(mvLX y )<<(N+K)).

[0279] (iv) Similarly, if it is necessary to convert MV(v x ,v y If you scale to a lower precision, you can apply the scaling process specified in Example 1(d).

[0280] (c) In one example, if MV(v x ,v y The accuracy of ) is lower / higher than MV(mvLX) x ,mvLX y If the accuracy is such that MV(v) is... x ,v y ) Scale to a finer / coarser precision. For example, MV(mvLX) x ,mvLX y If the precision is 1 / 16 of a pixel, then MV(v) x ,v y It is also scaled to 1 / 16 pixel precision.

[0281] (d) If it is necessary to transfer (v) x ,v y Shift right (i.e., scale to a lower precision) by N bits to obtain the result with respect to (mvLX) x ,mvLX y If the same precision is achieved, then v x =(v x +offset)>>N, v y =(v y +offset)>>N, where, for example, offset = 1<<(N-1).

[0282] (i) alternatively, v x =sign(v x )*((abs(v x ()+offset)>>N), v y =sign(v y )*((abs(v y )+offset)>>N).

[0283] (ii) Similarly, if it is necessary to convert MV(mvLX) x ,mvLX y If scaling to a higher precision is desired, the scaling process described above, as specified in Example 1(b), can be applied.

[0284] (e) In one example, it is suggested that the MV(v) derived in BIO be used. x ,v y ) scale and add it to the original MV(mvLX) of the current block / sub-block. x ,mvLX y (X = 0 or 1). The updated MV is calculated as: mvL0' x =-v x *(τ0 / (τ0+τ1))+mvL0 x ,mvL0' y =-v y *(τ0 / (τ0+τ1))+mvL0 y And mvL1' x =v x *(τ1 / (τ0+τ1))+mvL1 x ,mvL1' y =v y *(τ1 / (τ0+τ1))+mvL1 y .

[0285] (i) In one example, the updated MV is used for future motion prediction (such as in AMVP, Merge and Affine modes), deblocking, OBMC, etc.

[0286] (ii) Alternatively, the updated MV can only be used for motion prediction of its non-immediately following CU / PU in the decoding order.

[0287] (iii) Alternatively, the updated MV can only be used as a TMVP in AMVP, Merge or Affine mode.

[0288] (f) If it is necessary to transfer (v) x ,v y Shift right (i.e., scale to a lower precision) by N bits to obtain the result with respect to (mvLX) x ,mvLX y If the same precision is achieved, then v x =(v x +offset)>>(N+K), v y =(v y +offset)>>(N+K), where, for example, offset = 1<<(N+K–1). K is an integer, for example, K equals 1, 2, 3, -2, -1 or 0.

[0289] (i) alternatively, v x =sign(v x )*((abs(v x ()+offset)>>(N+K)), v y =sign(v y )*((abs(v y )+offset)>>(N+K)), where, for example, offset=1<<(N+K–1).

[0290] Example 2. Instead of considering the POC distance (e.g., in the calculation of τ0 and τ1 as described above), the scaling method for MV called in the BIO process can be simplified.

[0291] (a)mvL0' x =-v x / S0+mvL0 x ,,mvL0' y =-v y / S0+mvL0 y , and / or mvL1' x =v x / S1+mvL1 x ,mvL1' y =v y / S1+mvL1 y In one example, S0 and / or S1 are set to 2. In another example, it is called under certain conditions (such as τ0>0 and τ1>0).

[0292] (i) Alternatively, an offset can be added during the segmentation process. For example, mvL0' x =(-v x +offset0) / S0+mvL0 x ,mvL0' y =-(v y +offset0) / S0+mvL0 y , and / or mvL1' x =(v x +offset1) / S1+mvL1 x ,mvL1' y =(v y +offset1) / S1+mvL1 y .

[0293] In one example, offset0 is set to S0 / 2, and offset1 is set to S1 / 2.

[0294] (ii) In one example, mvL0' x =((-v x +1)>>1)+mvL0 x ,mvL0' y =(-(v) y +1)>>1)+mvL0 y , and / or mvL1' x =((v x +1)>>1)+mvL1 x ,mvL1' y =((v y +1)>>1)+mvL1 y .

[0295] (b)mvL0' x =-SF0*v x +mvL0 x ,mvL0' y =-v y *SF0+mvL0 y , and / or mvL1' x =-SF1*v x +mvL1 x ,mvL1' y =-SF1*v y +mvL1 y In one example, SF0 is set to 2, and / or SF1 is set to 1. In another example, it is called under certain conditions (such as τ0>0 and τ1<0 and τ0>|τ1|), as... Figure 25 As shown in (b).

[0296] (c)mvL0' x =SFACT0*v x +mvL0 x ,mvL0' y =SFACT0*v y +mvL0 y , and / or mvL1' x =SFACT1*v x +mvL1 x ,mvL1' y =SFACT1*v y +mvL1 y In one example, SFACT0 is set to 1, and / or SFACT1 is set to 2. In another example, it is called under certain conditions (such as τ0>0 and τ1<0 and τ0<|τ1|), as... Figure 25 As shown in (c).

[0297] Example 3. When τ0>0 and τ1>0, (v x ,v y The derivation of (mvLX) and (mvLX) x ,mvLX y Updates can be done together to maintain high accuracy.

[0298] (a) In one example, if it is necessary to (v) x ,v y Shift right (i.e., scale to a lower precision) by N bits to obtain the result with respect to (mvLX) x ,mvLX y If the same precision is achieved, then mvL0' x =((-v x +offset)>>(N+1))+mvL0 x ,mvL0' y =((-v y +offset)>>(N+1))+mvL0 y ,mvL1' x =((v x +offset)>>(N+1))+mvL1 x ,mvL1' y =((v y +offset)>>(N+1))+mvL1 y For example, offset = 1 < <N.

[0299] (b) In one example, if it is necessary to transfer (v) x ,v y Shift right (i.e., scale to a lower precision) by N bits to obtain the result with respect to (mvLX) x ,mvLX y If the same precision is achieved, then mvL0' x =((-v x +offset)>>(N+K+1))+mvL0 x ,mvL0' y =((-v y +offset)>>(N+K+1))+mvL0 y ,mvL1' x =((v x +offset)>>(N+K+1))+mvL1 x ,mvL1' y =((v y +offset)>>(N+K+1))+mvL1 yFor example, offset = 1 << (N + K). K is an integer, for example, K equals 1, 2, 3, -2, -1 or 0.

[0300] (c) Alternatively, mvL0' x =-sign(v x )*((abs(v x )+offset)>>(N+1))+mvL0 x ,mvL0' y =-sign(v y )*((abs(v y )+offset)>>(N+1))+mvL0 y ,mvL1' x =sign(v x )*((abs(v x )+offset)>>(N+1))+mvL1 x ,mvL1' y =sign(v y )*((abs(v y )+offset)>>(N+1))+mvL1 y .

[0301] (d) Alternatively, mvL0' x =-sign(v x )*((abs(v x )+offset)>>(N+K+1))+mvL0 x ,mvL0' y =-sign(v y )*((abs(v y )+offset)>>(N+K+1))+mvL0 y ,mvL1' x =sign(v x )*((abs(v x )+offset)>>(N+K+1))+mvL1 x ,mvL1' y =sign(v y )*((abs(v y )+offset)>>(N+K+1))+mvL1 y For example, offset =

[0302] 1 << (N+K). K is an integer, for example, K equals 1, 2, 3, -2, -1 or 0.

[0303] Example 4. The pruning operation can be further applied to updated MVs used in BIO and / or DMVR, or other types of encoding methods that may require MV updates.

[0304] (a) In one example, the updated MV is cropped in the same way as other traditional MVs, such as cropping within a specific range compared to the image boundaries.

[0305] (b) Alternatively, the updated MV is clipped to a specific range (or multiple ranges for different sub-blocks) compared to the MV used in the MC process. That is, the difference between the MV used in the MC and the updated MV is clipped to a certain range (or multiple ranges for different sub-blocks).

[0306] Example 5. The use of updated MVs can be restricted in BIO and / or other types of coding methods that may require updating MVs.

[0307] (a) In one example, the updated MV is used for future motion prediction (such as in AMVP, Merge, and / or affine modes), deblocking, OBMC, etc. Alternatively, the updated MV can be used in the first module, while the original MV can be used in the second module. For example, the first module is motion prediction, and the second module is deblocking.

[0308] (i) In one example, future motion prediction refers to motion prediction in the block to be encoded / decoded after the current block in the current picture or strip.

[0309] (ii) Alternatively, future motion prediction refers to motion prediction in images or strips to be encoded / decoded after the current image or strip.

[0310] (b) Alternatively, the updated MV can only be used for motion prediction of its non-immediately following CU / PU in the decoding order.

[0311] (c) The updated MV should not be applied to the motion prediction of the next CU / PU in the decoding order.

[0312] (d) Alternatively, the updated MV may only be used as a predictor for encoding subsequent pictures / slices / strips, such as TMVP in AMVP, and / or Merge and / or affine patterns.

[0313] (e) Alternatively, the updated MV may only be used as a predictor for encoding subsequent pictures / slices / strips, such as ATMVP and / or STMVP.

[0314] Example 6.In one example, a two-step inter-frame prediction process is proposed, wherein a first step is performed to generate some intermediate predictions (first prediction) based on motion information associated with the current block via signaling notification / derivation, and a second step is performed to derive the final prediction (second prediction) of the current block based on updated motion information that may depend on the intermediate predictions.

[0315] (a) In one example, the BIO process (i.e., using motion information notified / derived by signaling, which is used to generate the first prediction and the spatial gradient, temporal gradient and optical flow of each sub-block / pixel within the block) is used only to derive the updated MV as specified in Example 1 (and does not apply Equation (7) to generate the second prediction), and then the updated MV is used to perform motion compensation and generate the second prediction (i.e. the final prediction) of each sub-block / pixel within the block.

[0316] (b) In one example, memory bandwidth can be reduced by using an interpolation filter that is different from the interpolation filter of the inter-coded block that is not encoded in this way in the first or / and second steps.

[0317] (i) In one example, a shorter tap filter (such as a 6-tap filter, a 4-tap filter, or a bilinear filter) can be used.

[0318] (ii) Alternatively, filters (such as filter taps, filter coefficients) used in the first / second step can be predefined.

[0319] (iii) Alternatively, the filter taps selected for the first and / or second steps may depend on coding information such as block size / block shape (square, non-square, etc.) / strip type / prediction direction (unidirectional or bidirectional prediction or multiple hypotheses, forward or backward).

[0320] (iv) Alternatively, different blocks can select different filters for the first / second steps. In one example, one or more candidate sets of multiple filters can be predefined or signaled. Blocks can select from the candidate sets. The selected filter can be indicated by an index signaled, or it can be derived on the fly without signaling.

[0321] (c) In one example, only integer MV is used when generating the first prediction, and no interpolation filtering process is applied in the first step.

[0322] (i) In one example, the fraction MV is rounded to the nearest integer MV.

[0323] (1) If there is more than one nearest integer MV, round the fraction MV to the smaller nearest integer MV.

[0324] (2) If there is more than one nearest integer MV, round the fraction MV to the larger nearest integer MV.

[0325] (3) If there is more than one nearest integer MV, round the fraction MV to the nearest MV that is closer to zero.

[0326] (ii) In one example, the fraction MV is rounded to the nearest integer MV that is not less than the fraction MV.

[0327] (iii) In one example, the fraction MV is rounded to the nearest integer MV that is not greater than the fraction MV.

[0328] (d) The use of this method can be notified by signaling in SPS, PPS, strip header, CTU or CU or CTU group.

[0329] (e) The use of this method can also depend on encoding information such as block size / block shape (square, non-square, etc.) / strip type / prediction direction (unidirectional or bidirectional prediction or multiple hypotheses, forward or backward).

[0330] (i) In one example, this method can be automatically disabled under certain conditions, for example, when the current block is encoded in affine mode.

[0331] (ii) In one example, this approach can be applied automatically under certain conditions, such as when blocks are encoded with double predictions and the block size is greater than a threshold (e.g., more than 16 samples).

[0332] Example 7. In one example, it is proposed that the reference block (or prediction block) can be modified before calculating the temporal gradient in BIO, and the calculation of the temporal gradient is based on the modified reference block.

[0333] (a) In one example, the mean is removed for all reference blocks.

[0334] (i) For example, for a reference block X (X = 0 or 1), first, the mean (denoted by MeanX) is calculated for the block, and then MeanX is subtracted from each pixel in the reference block.

[0335] (ii) Alternatively, for different lists of reference images, it may be decided whether to remove the mean.

[0336] For example, for one reference block / sub-block, the mean is removed before calculating the time gradient, while for another reference block / sub-block, the mean is not removed.

[0337] (iii) Alternatively, different reference blocks (e.g., 3 or 4 reference blocks used in multi-hypothesis prediction) may be modified first.

[0338] (b) In one example, the mean is defined as the average of the selected samples in the reference block.

[0339] (c) In one example, all pixels in reference block X or a sub-block of reference block X are used to calculate MeanX.

[0340] (d) In one example, only a subset of pixels in the reference block X or a sub-block of the reference block are used to calculate MeanX. For example, only pixels from every second row / column are used.

[0341] (i) Alternatively, in the example, MeanX is calculated using only the pixels of every fourth row / column.

[0342] (ii) Alternatively, MeanX can be calculated using only the four corner pixels.

[0343] (iii) Alternatively, MeanX can be calculated using only the four corner pixels and the center pixel, for example, the pixel at position (W / 2, H / 2) (where W×H is the reference block size).

[0344] (e) In one example, the reference block can be filtered before it is used to derive the temporal gradient.

[0345] (i) In one example, a smoothing filter method can be applied to the reference block first.

[0346] (ii) In one example, the pixels at the block boundary are first filtered.

[0347] (iii) In one example, overlapping block motion compensation (OBMC) is applied first before deriving the time gradient.

[0348] (iv) In one example, illumination compensation (IC) is applied first before deriving the time gradient.

[0349] (v) In one example, weighted predictions are applied before deriving the time gradient.

[0350] (f) In one example, the time gradient is first calculated and then modified. For example, the time gradient is further subtracted from the difference between Mean0 and Mean1.

[0351] Example 8.In one example, the encoder may signal to the decoder in a Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Slice, CTU, or CU to notify whether to update the MV for the BIO coded block and / or to use the updated MV for future motion prediction and / or how to use the updated MV for future motion prediction.

[0352] Example 9. In one example, constraints are proposed to be added to the motion vectors used in the BIO process.

[0353] (a) In one example, (v x ,v y Constrained to a given range, -M x <v x <N x , and / or -M y <v y <N y M x N x M y N y It is a non-negative integer, and can be equal to, for example, 32.

[0354] (b) In one example, constrain the MV of the BIO-encoded sub-block / BIO-encoded block update to a given range, such as -M L0x <mvL0’ x <N L0x and / or -M L1x <mvL1’ x <N L1x ,-M L0y <mvL0’ y <N L0y and / or -M L1y <mvL1’ y <N L1y M L0x N L0x M L1x N L1x M L0y N L0y M L1y N L1y It is a non-negative integer, and can be equal to, for example, 1024, 2048, etc.

[0355] Example 10. This paper proposes other methods for updating the motion information (MV) (or including the motion information of the MV and / or reference image) for BIO, DMVR, FRUC, template matching, or content that needs to be deduced from the bitstream, where the use of the updated motion information may be constrained.

[0356] (a) In one example, even when updating motion information at the block level, updated and non-updated motion information can be stored differently for different sub-blocks. In one example, updated motion information for some sub-blocks can be stored, and non-updated motion information can be stored for the remaining sub-blocks.

[0357] (b) In one example, if the MV (or motion information) is updated at the sub-block / block level, the updated MV is stored only for the internal sub-blocks (i.e., sub-blocks not at the PU / CU / CTU boundary), and then, as Figure 26A and 26B As shown, it can be used for motion prediction, deblocking, OBMC, etc. Alternatively, the updated MV can be stored only for boundary sub-blocks.

[0358] (c) In one example, if the neighboring block and the current block are not in the same CTU or in the same region with a size such as 64×64 or 32×32, then the updated motion information from the neighboring block is not used.

[0359] (i) In one example, if the adjacent block and the current block are not in the same CTU or in the same area with a size such as 64×64 or 32×32, the adjacent block is considered "unavailable".

[0360] (ii) Alternatively, if the adjacent block and the current block are not in the same CTU or in the same region with a size such as 64×64 or 32×32, the current block uses motion information without an update process.

[0361] (d) In one example, if the adjacent block and the current block are not in the same CTU row or in the same row of a region with a size such as 64×64 or 32×32, the updated MV from the adjacent block is not used.

[0362] (i) In one example, if the adjacent block and the current block are not in the same CTU row or in the same row of a region with a size such as 64×64 or 32×32, the adjacent block is considered "unavailable".

[0363] (ii) Alternatively, if the adjacent block and the current block are not in the same CTU row or in the same row of a region with a size such as 64×64 or 32×32, the current block uses motion information without an update process.

[0364] (e) In one example, if the bottom row of the block is a CTU or the bottom row of an area with a size such as 64×64 or 32×32, the block's motion information is not updated.

[0365] (f) In one example, if the rightmost column of the block is a CTU or the rightmost column of an area with a size such as 64×64 or 32×32, the block's motion information is not updated.

[0366] (g) In one example, refined motion information from some neighboring CTUs or regions is used for the current CTU, and unrefined motion information from other neighboring CTUs or regions is used for the current CTU.

[0367] (i) In one example, refined motion information from the left CTU or the left region is used for the current CTU.

[0368] (ii) Alternatively, refined motion information from the top-left CTU or the top-left region may be used for the current CTU.

[0369] (iii) Alternatively, refined motion information from the upper CTU or the upper region may be used for the current CTU.

[0370] (iv) Alternatively, refined motion information from the upper right CTU or the upper right region may be used for the current CTU.

[0371] (v) In one example, the region has a size such as 64×64 or 32×32.

[0372] Example 11. In one example, it is proposed that different MVD precisions can be used in AF_INTER mode, and that the syntax elements can be signaled to indicate the MVD precision for each block / CU / PU. Precision sets comprising multiple different MVD precisions constituting a geometric sequence are allowed.

[0373] (a) In one example, {1 / 4,1,4} pixel MVD precision is allowed.

[0374] (b) In one example, {1 / 4, 1 / 2, 1, 2, 4} pixel MVD precision is allowed.

[0375] (c) In one example, {1 / 16, 1 / 8, 1 / 4} pixel MVD precision is allowed.

[0376] (d) Syntax elements exist under further conditions, such as when there are non-zero MVD components of block / CU / PU.

[0377] (e) In one example, MVD precision information is always communicated via signaling, regardless of the presence of any non-zero MVD components.

[0378] (f) Alternatively, for the 4 / 6 parameter AF_INTER mode, where 2 / 3 MVD is encoded, different MVD accuracies can be used for 2 / 3 MVD (1 MVD per control point in unidirectional prediction, 2 MVDs per control point in bidirectional prediction, i.e., 1 MVD per control point in each prediction direction), and the 2 / 3 control points are associated with different MVD accuracies. In this case, the 2 / 3 syntax elements can also be signaled to indicate the MVD accuracies.

[0379] (g) In one example, the method described in PCT / CN2018 / 091792 can be used to encode MVD precision in AF_INTER mode.

[0380] Example 12. In one example, it is proposed that if more than one DMVD method is performed on a block (e.g., PU), different decoder-side motion vector derivation (DMVD) methods such as BIO, DMVR, FRUC, and template matching work independently, i.e., the input of a DMVD method does not depend on the output of another DMVD method.

[0381] (a) In one example, in addition, a prediction block and / or a set of updated motion information (e.g., motion vectors and reference images for each prediction direction) are generated from multiple sets of motion information derived by multiple DMVD methods.

[0382] (b) In one example, motion compensation is performed using the motion information derived from each DMVD method, and these are averaged or weighted or filtered (e.g., by a median filter) to generate the final prediction.

[0383] (c) In one example, the motion information derived by all DMVD methods is averaged, weighted averaged, or filtered (e.g., through a median filter) to generate the final motion information. Alternatively, different priorities are assigned to different DMVD methods, and the motion information derived by the method with the highest priority is selected as the final motion information. For example, when BIO and DMVR are performed on the PU, the motion information generated by DMVR is used as the final motion information.

[0384] (d) In one example, for a PU, no more than N DMVD methods are allowed, where N>=1.

[0385] (i) Assign different priorities to different DMVD methods and execute the effective methods with the highest N priorities.

[0386] (e) DMVD methods are executed simultaneously. The updated MV of one DMVD method is not used as the starting input for the next DMVD method. For all DMVD methods, the unupdated MV is used as the starting point for the search. Alternatively, DMVD methods are executed in a cascaded manner. The updated MV of one DMVD method is used as the starting point for the search of the next DMVD method.

[0387] Other implementation plans

[0388] This section describes methods for refining and storing MVs for further use in BIO-coded blocks. The refined MVs can be used for motion vector prediction of subsequent blocks within the current strip / CTU line / slice, and / or for filtering (e.g., deblocking filter processing) and / or motion vector prediction of blocks located at different picture positions.

[0389] like Figure 32 As shown, the sub-block in reference block 0 points to the sub-block in reference block 1 (by (DMV) x DMV y The motion vector derived from the derivation of () is used to further improve the prediction of the current sub-block.

[0390] It is recommended to further refine the motion vectors of each sub-block by using motion vectors derived from BIO. The POC distance (e.g., absolute POC difference) between the LX reference image and the current image is represented as deltaPOCX, and (MVLX) x MVLX y ) and (MVLX x ',MVLX y ') represents the signaling notification and refined motion vector of the current sub-block, where X = 0 or 1. Then (MVLX x ',MVLX y The calculation is as follows:

[0391]

[0392] However, the above equation requires multiplication and division. To solve this problem, the derivation of the refined motion vector is simplified as follows:

[0393] MVL0′ x =MVL0 x -((DMV x +1)>>1)

[0394] MVL0′ y =MVL0 y -((DMV y +1)>>1)

[0395] MVL1′x = MVL1 x + ((DMV x + 1) >> 1)

[0396] MVL1' y = MVL1 y + ((DMV y + 1) >> 1)

[0397] In some embodiments, this method is only employed when predicting the current CU from the previous and the subsequent pictures, and thus it operates only in the random access (RA) configuration.

[0398] Example 13. The proposed method can be applied under certain conditions, such as block size, slice / picture / slice type.

[0399] (a) In one example, when the block size contains less than M*H samples, such as 16 or 32 or 64 luminance samples, the above method is not allowed.

[0400] (b) Alternatively, when the minimum dimension of the width or height of the block is less than or not greater than X, the above method is not allowed. In one example, X is set to 8.

[0401] (c) Alternatively, when the width of the block > th1 or >= th1 and / or the height of the block > th2 or >= th2, the above method is not allowed. In one example, X is set to 8.

[0402] (d) Alternatively, when the width of the block < th1 or <= th1 and / or the height of the block < th2 or <= th2, the above method is not allowed. In one example, X is set to 8.

[0403] Example 14. The above method can be applied at the sub-block level.

[0404] (a) In one example, the BIO update process, or the two-step inter-frame prediction process, or the temporal gradient derivation method described in Example 7 can be invoked for each sub-block.

[0405] (b) In one example, when the width or height or both the width and height of the block are greater than (or equal to) a threshold L, the block can be divided into multiple sub-blocks. Each sub-block is processed in the same manner as a normal coded block with the sub-block size.

[0406] Example 15. The threshold can be predefined or signaled at the SPS / PPS / picture / slice / slice level.

[0407] (a) Alternatively, the threshold can depend on certain coding information, such as block size, picture type, temporal layer index, etc.

[0408] The examples described above can be used in the context of the methods described below, such as methods 2700-3100, 3300-3600, and 3800-4200, which can be implemented at the video decoder.

[0409] Figure 27 A flowchart of an example method for video decoding is shown. Method 2700 includes, in step 2710, receiving a bitstream representation of the current block of video data.

[0410] Method 2700 includes, in step 2720, generating updated first and second reference motion vectors based on weighted sums of a first scaled motion vector and first and second scaled reference motion vectors, respectively. In some embodiments, the first scaled motion vector is generated by scaling the first motion vector to a target precision, and wherein the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to the target precision, respectively. In some embodiments, the first motion vector is derived based on a first reference motion vector from a first reference block and a second reference motion vector from a second reference block, and wherein the current block is associated with the first and second reference blocks.

[0411] In some embodiments, the indication of target precision is communicated via signaling in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), strip header, coding tree unit (CTU), or coding unit (CU).

[0412] In some embodiments, the first motion vector has a first precision, and the first and second reference motion vectors have reference precisions. In other embodiments, the first precision may be higher or lower than the reference precision. In other embodiments, the target precision may be set to the first precision, the reference precision, or a fixed (or predetermined) precision independent of the first precision and the reference precision.

[0413] In some embodiments, the first motion vector is derived based on bidirectional optical flow (BIO) refinement using first and second reference motion vectors.

[0414] Method 2700 includes, in step 2730, processing a bitstream representation based on updated first and second reference motion vectors to generate a current block. In some embodiments, the processing is based on bidirectional optical flow (BIO) thinning or decoder-side motion vector thinning (DMVR), and wherein the updated first and second reference motion vectors are clipped prior to processing.

[0415] In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement, and the updated first and second reference motion vectors are constrained to a predetermined range of values ​​prior to processing.

[0416] In some embodiments, the processing is based on bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC), or template matching techniques. In one example, updated first and second reference motion vectors are generated for internal sub-blocks that are not on the boundary of the current block. In another example, updated first and second reference motion vectors are generated for subsets of sub-blocks of the current block.

[0417] In some embodiments, the processing is based on at least two techniques, which may include bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC), or template matching. In one example, processing is performed for each of the at least two techniques to generate multiple result sets, which may be averaged or filtered to generate the current block. In another example, processing is performed in a cascaded manner for each of the at least two techniques to generate the current block.

[0418] Figure 28 A flowchart of an example method for video decoding is shown. Method 2800 includes, in step 2810, generating intermediate predictions for the current block based on first motion information associated with the current block. In some embodiments, generating intermediate predictions includes a first interpolation filtering process. In some embodiments, generating intermediate predictions is also based on signaling in a sequence parameter set (SPS), picture parameter set (PPS), coding tree unit (CTU), stripe header, coding unit (CU), or CTU group.

[0419] Method 2800 includes updating first motion information to second motion information in step 2820. In some embodiments, updating the first motion information includes refining using bidirectional optical flow (BIO).

[0420] Method 2800 includes, in step 2830, generating a final prediction for the current block based on intermediate predictions or second motion information. In some embodiments, generating the final prediction includes a second interpolation filtering process.

[0421] In some embodiments, the first interpolation filtering process uses a first filter set that differs from the second filter set used by the second interpolation filtering process. In some embodiments, at least one filter tap of the first or second interpolation filtering process is based on the dimension, prediction direction, or prediction type of the current block.

[0422] Figure 29 A flowchart of another example method for video decoding is shown. This example includes... Figure 28 The features and / or steps shown and described above are similar. At least some of these features and / or components may not be described separately in this section.

[0423] Method 2900 includes receiving a bitstream representation of the current block of video data in step 2910. In some embodiments, step 2910 includes receiving the bitstream representation from a memory location or buffer in a video encoder or decoder. In other embodiments, step 2910 includes receiving the bitstream representation at a video decoder via a wireless or wired channel. In other embodiments, step 2910 includes receiving the bitstream representation from a different module, unit, or processor, which may implement one or more methods as described in the embodiments of this document, but is not limited thereto.

[0424] Method 2900 includes, in step 2920, generating intermediate motion information based on motion information associated with the current block.

[0425] Method 2900 includes, in step 2930, generating updated first and second reference motion vectors based on first and second reference motion vectors, respectively. In some embodiments, the current block is associated with first and second reference blocks. In some embodiments, the first and second reference motion vectors are associated with first and second reference blocks, respectively.

[0426] Method 2900 includes, in step 2940, processing a bitstream representation based on intermediate motion information or updated first and second reference motion vectors to generate the current block.

[0427] In some embodiments of method 2900, the updated first and second reference motion vectors are generated based on weighted sums of the first scaled motion vector and the first and second scaled reference motion vectors, respectively. In some embodiments, the first motion vector is derived based on the first and second reference motion vectors, the first scaled motion vector is generated by scaling the first motion vector to a target precision, and the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to the target precision, respectively.

[0428] In some embodiments, the indication of target precision is communicated via signaling in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), strip header, coding tree unit (CTU), or coding unit (CU).

[0429] In some embodiments, the first motion vector has a first precision, and the first and second reference motion vectors have reference precisions. In other embodiments, the first precision may be higher or lower than the reference precision. In other embodiments, the target precision may be set to the first precision, the reference precision, or a fixed (or predetermined) precision independent of the first precision and the reference precision.

[0430] In some embodiments, the first motion vector is derived based on bidirectional optical flow (BIO) refinement using first and second reference motion vectors.

[0431] In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement, and the updated first and second reference motion vectors are constrained to a predetermined range of values ​​prior to processing.

[0432] In some embodiments, the processing is based on bidirectional optical flow (BIO) thinning or decoder-side motion vector thinning (DMVR), wherein the updated first and second reference motion vectors are clipped prior to processing.

[0433] In some embodiments, the processing is based on bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC), or template matching techniques. In one example, updated first and second reference motion vectors are generated for internal sub-blocks that are not on the boundary of the current block. In another example, updated first and second reference motion vectors are generated for subsets of sub-blocks of the current block.

[0434] In some embodiments, the processing is based on at least two techniques, which may include bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC), or template matching. In one example, processing is performed for each of the at least two techniques to generate multiple result sets, which may be averaged or filtered to generate the current block. In another example, processing is performed in a cascaded manner for each of the at least two techniques to generate the current block.

[0435] Figure 30 A flowchart of an example method for video decoding is shown. Method 3000 includes, in step 3010, generating intermediate predictions for the current block based on first motion information associated with the current block. In some embodiments, generating intermediate predictions includes a first interpolation filtering process. In some embodiments, generating intermediate predictions is also based on signaling in a sequence parameter set (SPS), picture parameter set (PPS), coding tree unit (CTU), stripe header, coding unit (CU), or CTU group.

[0436] Method 3000 includes updating first motion information to second motion information in step 3020. In some embodiments, updating the first motion information includes refining using bidirectional optical flow (BIO).

[0437] Method 3000 includes, in step 3030, generating a final prediction for the current block based on intermediate predictions or second motion information. In some embodiments, generating the final prediction includes a second interpolation filtering process.

[0438] In some embodiments, the first interpolation filtering process uses a first filter set that differs from the second filter set used by the second interpolation filtering process. In some embodiments, at least one filter tap of the first or second interpolation filtering process is based on the dimension, prediction direction, or prediction type of the current block.

[0439] Figure 31 A flowchart of another example method for video decoding is shown. This example includes methods similar to those described above. Figure 30 The features and / or steps shown are similar to some of the features and / or steps described herein. At least some of these features and / or components may not be described individually in this section.

[0440] Method 3100 includes, in step 3110, receiving a bitstream representation of the current block of video data. In some embodiments, step 3110 includes receiving the bitstream representation from a memory location or buffer in a video encoder or decoder. In other embodiments, step 3110 includes receiving the bitstream representation at a video decoder via a wireless or wired channel. In other embodiments, step 3110 includes receiving the bitstream representation from a different module, unit, or processor that may implement one or more methods as described in the embodiments herein, but is not limited thereto.

[0441] Method 3100 includes, in step 3120, generating intermediate motion information based on motion information associated with the current block.

[0442] Method 3100 includes, in step 3130, generating updated first and second reference motion vectors based on first and second reference motion vectors, respectively. In some embodiments, the current block is associated with first and second reference blocks. In some embodiments, the first and second reference motion vectors are associated with first and second reference blocks, respectively.

[0443] Method 3100 includes, in step 3140, processing a bitstream representation based on intermediate motion information or updated first and second reference motion vectors to generate the current block.

[0444] In some embodiments of method 3100, the updated first and second reference motion vectors are generated based on weighted sums of the first scaled motion vector and the first and second scaled reference motion vectors, respectively. In some embodiments, the first motion vector is derived based on the first and second reference motion vectors, the first scaled motion vector is generated by scaling the first motion vector to a target precision, and the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to the target precision, respectively.

[0445] In some embodiments, the indication of target precision is communicated via signaling in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), strip header, coding tree unit (CTU), or coding unit (CU).

[0446] In some embodiments, the first motion vector has a first precision, and the first and second reference motion vectors have reference precisions. In other embodiments, the first precision may be higher or lower than the reference precision. In other embodiments, the target precision may be set to the first precision, the reference precision, or a fixed (or predetermined) precision independent of the first precision and the reference precision.

[0447] In some embodiments, the first motion vector is derived based on bidirectional optical flow (BIO) refinement using first and second reference motion vectors.

[0448] In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement, and the updated first and second reference motion vectors are constrained to a predetermined range of values ​​prior to processing.

[0449] In some embodiments, the processing is based on bidirectional optical flow (BIO) thinning or decoder-side motion vector thinning (DMVR), wherein the updated first and second reference motion vectors are clipped prior to processing.

[0450] In some embodiments, the processing is based on bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC), or template matching techniques. In one example, updated first and second reference motion vectors are generated for internal sub-blocks that are not on the boundary of the current block. In another example, updated first and second reference motion vectors are generated for subsets of sub-blocks of the current block.

[0451] In some embodiments, the processing is based on at least two techniques, which may include bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC), or template matching. In one example, processing is performed for each of the at least two techniques to generate multiple result sets, which may be averaged or filtered to generate the current block. In another example, processing is performed in a cascaded manner for each of the at least two techniques to generate the current block.

[0452] Figure 33A flowchart of an example method for video decoding is shown. Method 3300 includes, in step 3310, generating intermediate predictions for the current block based on first motion information associated with the current block. In some embodiments, generating intermediate predictions includes a first interpolation filtering process. In some embodiments, generating intermediate predictions is also based on signaling in a sequence parameter set (SPS), picture parameter set (PPS), coding tree unit (CTU), stripe header, coding unit (CU), or CTU group.

[0453] Method 3300 includes updating first motion information to second motion information in step 3320. In some embodiments, updating the first motion information includes refining using bidirectional optical flow (BIO).

[0454] Method 3300 includes, in step 3330, generating a final prediction for the current block based on intermediate predictions or second motion information. In some embodiments, generating the final prediction includes a second interpolation filtering process.

[0455] In some embodiments, the first interpolation filtering process uses a first filter set that differs from the second filter set used by the second interpolation filtering process. In some embodiments, at least one filter tap of the first or second interpolation filtering process is based on the dimension, prediction direction, or prediction type of the current block.

[0456] Figure 34 A flowchart of another example method for video decoding is shown. This example includes methods similar to those described above. Figure 33 The features and / or steps shown are similar to some of the features and / or steps described herein. At least some of these features and / or components may not be described individually in this section.

[0457] Method 3400 includes, in step 3410, receiving a bitstream representation of the current block of video data. In some embodiments, step 3410 includes receiving the bitstream representation from a memory location or buffer in a video encoder or decoder. In other embodiments, step 3410 includes receiving the bitstream representation at a video decoder via a wireless or wired channel. In other embodiments, step 3410 includes receiving the bitstream representation from a different module, unit, or processor, which may implement one or more methods as described in the embodiments of this document, but is not limited thereto.

[0458] Method 3400 includes, in step 3420, generating intermediate motion information based on motion information associated with the current block.

[0459] Method 3400 includes, in step 3430, generating updated first and second reference motion vectors based on first and second reference motion vectors, respectively. In some embodiments, the current block is associated with first and second reference blocks. In some embodiments, the first and second reference motion vectors are associated with first and second reference blocks, respectively.

[0460] Method 3400 includes, in step 3440, processing a bitstream representation based on intermediate motion information or updated first and second reference motion vectors to generate the current block.

[0461] In some embodiments of method 3400, the updated first and second reference motion vectors are generated based on weighted sums of the first scaled motion vector and the first and second scaled reference motion vectors, respectively. In some embodiments, the first motion vector is derived based on the first and second reference motion vectors, the first scaled motion vector is generated by scaling the first motion vector to a target precision, and the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to the target precision, respectively.

[0462] In some embodiments, the indication of target precision is communicated via signaling in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), strip header, coding tree unit (CTU), or coding unit (CU).

[0463] In some embodiments, the first motion vector has a first precision, and the first and second reference motion vectors have reference precisions. In other embodiments, the first precision may be higher or lower than the reference precision. In other embodiments, the target precision may be set to the first precision, the reference precision, or a fixed (or predetermined) precision independent of the first precision and the reference precision.

[0464] In some embodiments, the first motion vector is derived based on bidirectional optical flow (BIO) refinement using first and second reference motion vectors.

[0465] In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement, and the updated first and second reference motion vectors are constrained to a predetermined range of values ​​prior to processing.

[0466] In some embodiments, the processing is based on bidirectional optical flow (BIO) thinning or decoder-side motion vector thinning (DMVR), wherein the updated first and second reference motion vectors are clipped prior to processing.

[0467] In some embodiments, the processing is based on bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC), or template matching techniques. In one example, updated first and second reference motion vectors are generated for internal sub-blocks that are not on the boundary of the current block. In another example, updated first and second reference motion vectors are generated for subsets of sub-blocks of the current block.

[0468] In some embodiments, the processing is based on at least two techniques, which may include bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC), or template matching. In one example, processing is performed for each of the at least two techniques to generate multiple result sets, which may be averaged or filtered to generate the current block. In another example, processing is performed in a cascaded manner for each of the at least two techniques to generate the current block.

[0469] Figure 35 A flowchart of an example method for video decoding is shown. Method 3500 includes, in step 3510, generating an updated reference block for the bitstream representation of the current block by modifying a reference block associated with the current block.

[0470] In some embodiments, method 3500 further includes the step of filtering the reference block using a smoothing filter.

[0471] In some embodiments, method 3500 further includes a step of filtering pixels at the block boundaries of the reference block.

[0472] In some embodiments, method 3500 further includes the step of applying Overlapping Block Motion Compensation (OBMC) to a reference block.

[0473] In some embodiments, method 3500 further includes the step of applying illumination compensation (IC) to a reference block.

[0474] In some embodiments, method 3500 further includes the step of applying a weighted prediction to the reference block.

[0475] Method 3500 includes, in step 3520, calculating the temporal gradient for bidirectional optical flow (BIO) motion refinement based on the updated reference block.

[0476] Method 3500 includes, in step 3530, performing a transformation including BIO motion refinement between a bitstream representation and a current block based on a temporal gradient. In some embodiments, the transformation generates the current block from the bitstream representation (e.g., it may be implemented in a video decoder). In other embodiments, the transformation generates a bitstream representation from the current block (e.g., it may be implemented in a video encoder).

[0477] In some embodiments, method 3500 further includes calculating the mean of a reference block; and subtracting the mean from each pixel of the reference block. In one example, the mean is calculated based on all pixels of the reference block. In another example, the mean is calculated based on all pixels in a sub-block of the reference block.

[0478] In some embodiments, the mean is calculated as a subset of the pixels of a reference block (in other words, not all pixels). In one example, the subset of pixels includes the pixels in every fourth row or column of the reference block. In another example, the subset of pixels includes the four corner pixels. In yet another example, the subset of pixels includes the four corner pixels and the center pixel.

[0479] Figure 36 A flowchart of another example method for video decoding is shown. This example includes methods similar to those described above. Figure 35 The features and / or steps shown are similar to some of the features and / or steps described herein. At least some of these features and / or components may not be described individually in this section.

[0480] Method 3600 includes, in step 3610, generating a temporal gradient for bidirectional optical flow (BIO) motion refinement for the bitstream representation of the current block.

[0481] Method 3600 includes, in step 3620, generating an updated temporal gradient by subtracting the difference between a first mean and a second mean from the temporal gradient, wherein the first mean is the mean of a first reference block, the second mean is the mean of a second reference block, and the first and second reference blocks are associated with the current block.

[0482] In some embodiments, the mean is based on all pixels of the corresponding reference block (e.g., the first mean is calculated as the mean of all pixels of the first reference block). In another example, the mean is calculated based on all pixels of the sub-blocks of the corresponding reference block.

[0483] In some embodiments, the mean is based on a subset of the pixels of the corresponding reference block (in other words, not all pixels). In one example, the subset of pixels includes the pixels in every fourth row or column of the corresponding reference block. In another example, the subset of pixels includes the four corner pixels. In yet another example, the subset of pixels includes the four corner pixels and the center pixel.

[0484] Method 3600 includes, in step 3630, performing a transformation including BIO motion refinement between the bitstream representation and the current block based on an updated temporal gradient. In some embodiments, the transformation generates the current block from the bitstream representation (e.g., it may be implemented in a video decoder). In other embodiments, the transformation generates the bitstream representation from the current block (e.g., it may be implemented in a video encoder).

[0485] Figure 38 A flowchart of an example method for video processing is shown. The method 3800 includes: in step 3810, determining the original motion information of the current block; in step 3820, scaling the original motion vector and the derived motion vector derived from the original motion vector to the same target precision; in step 3830, generating an updated motion vector from the scaled original and derived motion vectors; and in step 3840, performing a conversion between the current block and a bitstream representation of the video including the current block based on the updated motion vector.

[0486] Figure 39 A flowchart of an example method for video processing is shown. The method 3900 includes: in step 3910, determining the original motion information of the current block; in step 3920, updating the original motion vector of the original motion information of the current block based on a thinning method; in step 3930, cropping the updated motion vector to a range; and in step 3940, performing a conversion between the current block and a bitstream representation of the video including the current block based on the cropped updated motion vector.

[0487] Figure 40 A flowchart of an example method for video processing is shown. The method 4000 includes: in step 4010, determining raw motion information associated with the current block; in step 4020, generating updated motion information based on a specific prediction mode; and in step 4030, performing a conversion between the current block and a bitstream representation of video data including the current block based on the updated motion information, wherein the specific prediction mode includes one or more of bidirectional optical flow (BIO) thinning, decoder-side motion vector thinning (DMVR), frame rate upconversion (FRUC) techniques, or template matching techniques.

[0488] Figure 41 A flowchart of an example method for video processing is shown. The method 4100 includes: in step 4110, determining the MVD precision of the current block processed in an affine mode from a set of motion vector difference (MVD) precisions; and in step 4120, performing a conversion between the current block and a bitstream representation of the video including the current block based on the determined MVD precision.

[0489] Figure 42 A flowchart of an example method for video processing is shown. The method 4200 includes: in step 4210, determining unupdated motion information associated with the current block; in step 4220, updating the unupdated motion information based on multiple decoder-side motion vector derivation (DMVD) methods to generate updated motion information for the current block; and in step 4230, performing a conversion between the current block and a bitstream representation of the video including the current block based on the updated motion information.

[0490] 7. Example implementation of the disclosed technology

[0491] Figure 37 This is a block diagram of a video processing apparatus 3700. Apparatus 3700 can be used to implement one or more methods described herein. Apparatus 3700 can be embodied in a smartphone, tablet, computer, Internet of Things (IoT) receiver, etc. Apparatus 3700 may include one or more processors 3702, one or more memories 3704, and video processing hardware 3706. Processor 3702 can be configured to implement one or more methods described herein (including, but not limited to, methods 2700-3100, 3300-3600, and 3800-4200). Memory (one or more) Memory 3704 can be used to store data and code for implementing the methods and techniques described herein. Video processing hardware 3706 can be used to implement some of the techniques described in this document in hardware circuitry.

[0492] In some embodiments, the video encoding method may use methods such as those described above. Figure 37 The device described is implemented on a hardware platform.

[0493] The various embodiments and techniques described throughout this document can be described using the following clause-based format.

[0494] 1.1 A video processing method, comprising:

[0495] Determine the original motion information of the current block;

[0496] The original motion vector of the original motion information and the derived motion vector derived from the original motion vector are scaled to the same target accuracy;

[0497] Generate updated motion vectors from the scaled original and derived motion vectors; and

[0498] Based on the updated motion vectors, perform a conversion between the current block and the bitstream representation of the video including the current block.

[0499] 1.2. The method as in Example 1.1, wherein the original motion vector has a first precision, the derived motion vector has a second precision different from the first precision, and the target precision is set to a higher or lower precision between the first and second precisions.

[0500] 1.3. As in Example 1.1, where the target precision is set to a fixed precision.

[0501] 1.4. As in Example 1.1, where the target accuracy is higher than the accuracy of the original motion vector.

[0502] 1.5. As in Example 1.4, where the original motion vector is scaled to:

[0503] mvLX' x =sign(mvLX) x )*(abs(mvLX x )< <N),

[0504] mvLX' y =sign(mvLX) y )*(abs(mvLX y )< <N),

[0505] Where (mvLX) x ,mvLX y ) is the original motion vector, (mvLX' x ,mvLX' y ) is the scaled original motion vector, the function sign(.) returns the sign of the input parameter, the function abs(.) returns the absolute value of the input parameter, N = log2(curr_mv_precision / targ_mv_precision), where curr_mv_precision is the precision of the original motion vector, and targ_mv_precision is the precision of the derived motion vector, which is used as the target precision.

[0506] 1.6. As in Example 1.1, where the target accuracy is the same as the accuracy of the original motion vector.

[0507] 1.7. The method as in Example 1.1, wherein the original motion vector has a first precision, the derived motion vector has a second precision different from the first precision, and the target precision is set to the first precision.

[0508] 1.8. As in Example 1.7, where the derived motion vector is scaled to the right by N to achieve the target accuracy:

[0509] v' x =(v x +offset)>>N,v' y =(v y +offset)>>N; or

[0510] v' x =sign(v x )*((abs(v x )+offset)>>N),v' y =sign(v y )*((abs(v y )+offset)

[0511] >>N)

[0512] Where (v x ,v y (v') is the derived motion vector. x ,v' y ) is the scaled derivation of the motion vector.

[0513] offset is the deflection used to derive motion vectors to achieve target accuracy.

[0514] The function `sign(.)` returns the sign of the input argument, and the function `abs(.)` returns the absolute value of the input argument.

[0515] N = log2(curr_mv_precision / targ_mv_precision), where curr_mv_precision is the first precision and targ_mv_precision is the second precision.

[0516] 1.9. As in Example 1.1, wherein the generation of the scaling and updating motion vectors is performed as follows:

[0517] mvL0' x =-v x / S0+mvL0 x ,mvL0' y =-v y / S0+mvL0 y ; and / or

[0518] mvL1' x =v x / S1+mvL1 x ,mvL1' y =v y / S1+mvL1 y

[0519] Among them, (mvL0 x ,mvL0 y ) and (mvL1 x mvL1 y ) is the original motion vector, (mvL0' x ,mvL0' y ) and (mvL1' x ,mvL1' y ) is the updated motion vector, (v x ,v y ) is the derived motion vector, and S0 and S1 are scaling factors.

[0520] 1.10. As in Example 1.1, wherein the generation of the scaling and updating motion vectors is performed as follows:

[0521] mvL0' x =(-v x +offset0) / S0+mvL0 x ,mvL0' y =-(v y +offset0) / S0+

[0522] mvL0 y , and / or

[0523] mvL1' x =(v x +offset1) / S1+mvL1 x ,mvL1' y =(v y +offset1) / S1+mvL1 y

[0524] Among them, (mvL0 x ,mvL0 y ) and (mvL1 x mvL1 y ) is the original motion vector, (mvL0' x ,mvL0' y ) and (mvL1' x ,mvL1' y ) is the updated motion vector, (v x ,v y ) is the derived motion vector, offset0 and offset1 are offsets, and S0 and S1 are scaling factors.

[0525] 1.11. As in Example 1.1, wherein the generation of the scaling and updating motion vectors is performed as follows:

[0526] mvL0' x =((-v x +1)>>1)+mvL0 x ,mvL0' y =(-(v) y +1)>>1)+mvL0 y ;and /

[0527] or

[0528] mvL1' x =((v x +1)>>1)+mvL1 x ,mvL1' y=((v y +1)>>1)+mvL1 y

[0529] Among them, (mvL0 x ,mvL0 y ) and (mvL1 x mvL1 y ) is the original motion vector, (mvL0' x ,mvL0' y ) and (mvL1' x ,mvL1' y ) is the updated motion vector, and (v x ,v y ) is the derived motion vector.

[0530] 1.12. The method of any one of Examples 1.9-1.11, wherein the generation of the scaling and updated motion vector is performed when τ0>0 and τ1>0, wherein τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current), and wherein POC(current), POC(Ref0), and POC(Ref1) are the image sequence counts of the current block, the first reference block, and the second reference block, respectively.

[0531] 1.13. As in Example 1.1, wherein the generation of the scaling and updating motion vectors is performed as follows:

[0532] mvL0' x =-SF0*v x +mvL0 x ,mvL0' y =-v y *SF0+mvL0 y ; and / or

[0533] mvL1' x =-SF1*v x +mvL1 x ,mvL1' y =-SF1*v y +mvL1 y

[0534] Among them, (mvL0 x ,mvL0 y ) and (mvL1 x mvL1 y ) is the original motion vector, (mvL0' x ,mvL0' y ) and (mvL1' x ,mvL1'y ) is the updated motion vector, (v x ,v y ) is the derived motion vector, and SF0 and SF1 are scaling factors.

[0535] 1.14. As in Example 1.13, wherein when τ0>0, τ1<0 and τ0>|τ1|, the generation of the scaling and updating motion vector is performed, where τ0=POC(current)-POC(Ref0), τ1=POC(Ref1)-POC(current), and where POC(current), POC(Ref0) and POC(Ref1) are the image sequence counts of the current block, the first reference block and the second reference block, respectively.

[0536] 1.15. As in Example 1.1, wherein the generation of the scaling and updating motion vectors is performed as follows:

[0537] mvL0' x =SFACT0*v x +mvL0 x ,mvL0' y =SFACT0*v y +mvL0 y , and / or

[0538] mvL1' x =SFACT1*v x +mvL1 x ,mvL1' y =SFACT1*v y +mvL1 y

[0539] Among them, (mvL0 x ,mvL0 y ) and (mvL1 x mvL1 y ) is the original motion vector, (mvL0' x ,mvL0' y ) and (mvL1' x ,mvL1' y ) is the updated motion vector, (v x ,v y ) is the derived motion vector, and SFACT0 and SFACT1 are scaling factors.

[0540] 1.16. As in Example 1.15, wherein when τ0>0, τ1<0 and τ0<|τ1|, the generation of the scaling and updating motion vector is performed, where τ0=POC(current)-POC(Ref0), τ1=POC(Ref1)-POC(current), and where POC(current), POC(Ref0) and POC(Ref1) are the image sequence counts of the current block, the first reference block and the second reference block, respectively.

[0541] 1.17. As in Example 1.1, when τ0>0, τ1>0 and τ0>|τ1|, the derivation of the derived motion vector and the generation of the updated motion vector are performed together, where τ0=POC(current)-POC(Ref0), τ1=POC(Ref1)-POC(current), and where POC(current), POC(Ref0) and POC(Ref1) are the image sequence counts of the current block, the first reference block and the second reference block, respectively.

[0542] 1.18. As in Example 1.17, wherein the generation of the scaled and updated motion vector is performed when the derived motion vector is shifted right by N to achieve the target accuracy:

[0543] mvL0' x =((-v x +offset)>>(N+1))+mvL0 x ,mvL0' y =((-v y +offset1)>>

[0544] (N+1))+mvL0 y ,mvL1' x =((v x +offset)>>(N+1))+mvL1 x ,mvL1' y =((v y +

[0545] offset2)>>(N+1))+mvL1 y ,

[0546] Among them, (mvL0 x ,mvL0 y ) and (mvL1 x mvL1 y ) is the original motion vector, (mvL0' x ,mvL0' y ) and (mvL1' x ,mvL1' y ) is the updated motion vector, (vx ,v y ) is the derived motion vector, offset1 and offset2 are offsets, N = log2(curr_mv_precision / targ_mv_precision), where curr_mv_precision is the precision of the original motion vector and targ_mv_precision is the precision of the derived motion vector.

[0547] 1.19. The method of Example 1.17, wherein the original motion vector has a first precision, the derived motion vector has a second precision different from the first precision, and the original motion vector is shifted left by N to achieve the target precision as the second precision.

[0548] 1.20. The method of Example 1.17, wherein the original motion vector is shifted left by K and the derived motion vector is shifted right by NK to achieve the target accuracy.

[0549] 1.21. The method as in Example 1.17, wherein the generation of the scaling and updating motion vectors is performed as follows:

[0550] mvL0' x =-sign(v x )*((abs(v x )+offset0)>>(N+1))+mvL0 x ,

[0551] mvL0' y =-sign(v y )*((abs(v y )+offset0)>>(N+1))+mvL0 y ,

[0552] mvL1' x =sign(v x )*((abs(v x )+offset1)>>(N+1))+mvL1 x ,

[0553] mvL1' y =sign(v y )*((abs(v y )+offset1)>>(N+1))+mvL1 y

[0554] Among them, (mvL0 x ,mvL0 y ) and (mvL1 x mvL1y ) is the original motion vector, (mvL0' x ,mvL0' y ) and (mvL1' x ,mvL1' y ) is the updated motion vector, (v x ,v y ) is the derived motion vector, offset0 and offset1 are offsets, the function sign(.) returns the sign of the input parameter, the function abs(.) returns the absolute value of the input parameter, N = log2(curr_mv_precision / targ_mv_precision), where curr_mv_precision is the precision of the original motion vector and targ_mv_precision is the precision of the derived motion vector.

[0555] 1.22. The method of Example 1.1, wherein updating the first and second reference motion vectors includes refining using bidirectional optical flow (BIO).

[0556] 1.23. A method as in any of Examples 1.1-1.22, where the method is not applied if a specific condition is met in the current block.

[0557] 1.24. The method of Example 1.23, wherein a specific condition specifies at least one of the following: the size of the current block, the strip type of the current block, the picture type of the current block, and the slice type of the current block.

[0558] 1.25. The method of Example 1.23, wherein a specific condition specifies that the number of samples contained in the current block is less than a first threshold.

[0559] 1.26. The method of Example 1.23, wherein a specific condition specifies that the minimum dimensions of the width and height of the current block are less than or not greater than a second threshold.

[0560] 1.27. The method of Example 1.23, wherein specific conditions specify that the width of the current block is less than or not greater than a third threshold, and / or the height of the current block is less than or not greater than a fourth threshold.

[0561] 1.28. The method of Example 1.23, wherein specific conditions specify that the width of the current block is greater than or not less than a third threshold, and / or the height of the current block is greater than or not less than a fourth threshold.

[0562] 1.29. The method of Example 1.23, wherein the method is applied at the sub-block level if the width and / or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

[0563] 1.30. The method of Example 1.29, wherein the current block is divided into multiple sub-blocks, and each of the multiple sub-blocks further undergoes a bidirectional optical flow (BIO) process in the same manner as a normal coded block having a size equal to that of the sub-block.

[0564] 1.31. The method as in any of Examples 1.25-1.29, wherein each of the first to fifth thresholds is predefined or signaled at the Sequence Parameter Set (SPS) level, or Picture Parameter Set (PPS) level, or Picture level, or Strip level, or Piece level.

[0565] 1.32. The method of Example 1.31, wherein each of the first to fifth thresholds is defined based on encoded information including at least one of block size, image type and time layer index.

[0566] 1.33. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any one of Examples 1.1 to 1.32.

[0567] 1.34. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing methods as described in any one of Examples 1.1 to 1.32.

[0568] 2.1. A video processing method, comprising:

[0569] Determine the original motion information of the current block;

[0570] The original motion vector of the original motion information of the current block is updated based on the refinement method;

[0571] Clip the updated motion vectors to a range; and

[0572] Based on the updated motion vectors from the cropped block, a conversion is performed between the current block and the bitstream representation of the video including the current block.

[0573] 2.2. The method as described in Example 2.1, wherein the refinement method includes bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate upconversion (FRUC), or template matching.

[0574] 2.3. The method as described in Example 2.1, wherein the updated motion vector is clipped to the same range allowed by the original motion vector.

[0575] 2.4. The method as described in Example 2.1, wherein the difference between the updated motion vector and the original motion vector is clipped to the same range or to different ranges for different sub-blocks.

[0576] 2.5. The method as described in Example 2.1, wherein the refinement method includes bidirectional optical flow (BIO) refinement, and the motion vector derived from the original motion vector in the BIO refinement is constrained to a first range, as follows:

[0577] -M x <v x <N x , and / or -M y <v y <N y ,

[0578] Where (v x ,v y ) is the derived motion vector, and M x N x M y N y It is a non-negative integer.

[0579] 2.6. The method as in Example 2.1, wherein the refinement method includes bidirectional optical flow (BIO) refinement and constrains the updated motion vector to a second range, as follows:

[0580] -M L0x <mvL0’ x <N L0x , and / or

[0581] -M L2.1x <mvL2.1’ x <N L2.1x , and / or

[0582] -M L0x <mvL0’ x <N L0x , and / or

[0583] -M L2.1y <mvL2.1’ y <N L2.1y

[0584] Where (mvL0') x ,mvL0' y ) and (mvL2.1' x ,mvL2.1' y ) is the updated motion vector from different reference lists, and M L0x N L0x M L2.1x N L2.1x M L0y N L0y M L2.1y N L2.1yIt is a non-negative integer.

[0585] 2.7. The method as described in any one of Examples 2.1-2.6, wherein the method is not applied if the current block satisfies a specific condition.

[0586] 2.8. The method as described in Example 2.7, wherein the specific condition specifies at least one of the following: the size of the current block, the strip type of the current block, the image type of the current block, and the slice type of the current block.

[0587] 2.9. The method as described in Example 2.7, wherein the specific condition specifies that the number of samples contained in the current block is less than a first threshold.

[0588] 2.10. The method as described in Example 2.7, wherein the specific condition specifies that the minimum dimensions of the width and height of the current block are less than or not greater than a second threshold.

[0589] 2.11. The method as described in Example 2.7, wherein the specific condition specifies that the width of the current block is less than or not greater than a third threshold, and / or the height of the current block is less than or not greater than a fourth threshold.

[0590] 2.12. The method as described in Example 2.7, wherein the specific condition specifies that the width of the current block is greater than or not less than a third threshold, and / or the height of the current block is greater than or not less than a fourth threshold.

[0591] 2.13. The method as described in Example 2.7, wherein the method is applied at the sub-block level if the width and / or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

[0592] 2.14. The method as described in Example 2.13, wherein the current block is divided into a plurality of sub-blocks, and each of the plurality of sub-blocks further undergoes a bidirectional optical flow (BIO) process in the same manner as a normal coded block having a size equal to that of the sub-block.

[0593] 2.15. The method as described in any one of Examples 2.9-2.13, wherein each of the first to fifth thresholds is predefined or signaled at the sequence parameter set SPS level, or the picture parameter set PPS level, or the picture level, or the strip level or the slice level.

[0594] 2.16. The method as described in Example 2.15, wherein each of the first to fifth thresholds is defined based on encoded information including at least one of block size, image type, and temporal layer index.

[0595] 2.17. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any one of Examples 2.1 to 2.16.

[0596] 2.18. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the methods as described in any one of Examples 2.1 to 2.16.

[0597] 3.1. A video processing method, comprising:

[0598] Determine the original motion information associated with the current block;

[0599] Generate updated motion information based on specific prediction patterns; and

[0600] Based on the updated motion information, a conversion is performed between the current block and a bitstream representation of the video data including the current block, wherein the specific prediction mode includes one or more of bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate upconversion (FRUC) technique, or template matching technique.

[0601] 3.2. The method as described in Example 3.1, wherein the updated motion information includes updated motion vectors.

[0602] 3.3. The method as described in Example 3.1, wherein the updated motion vector is used for motion prediction for encoding subsequent video blocks; or the updated motion vector is used for filtering or Overlapping Block Motion Compensation (OBMC).

[0603] 3.4. The method as described in Example 3.3, wherein the updated motion vector is used for motion prediction in Advanced Motion Vector Prediction (AMVP) mode, Merge mode, and / or affine mode.

[0604] 3.5. The method as described in Example 3.3, wherein the filtering includes deblocking filtering.

[0605] 3.6. The method as described in any one of Examples 3.1-3.5, wherein the updated motion information is used for the first module and the original motion information is used for the second module.

[0606] 3.7. The method as described in Example 3.6, wherein the first module is a motion prediction module and the second module is a deblocking module.

[0607] 3.8. The method as described in any one of Examples 3.2-3.7, wherein the motion prediction is used to process blocks following the current block in the current image or strip.

[0608] 3.9. The method as described in any one of Examples 3.2-3.7, wherein the motion prediction is used to process an image or strip to be processed after the current image or strip including the current block.

[0609] 3.10. The method as described in any one of Examples 3.1-3.9, wherein the updated motion vector is used only for motion information prediction of coding units (CUs) or prediction units (PUs) that do not immediately follow the current block in the processing order.

[0610] 3.11. The method as described in any one of Examples 3.1 to 3.10, wherein the updated motion vector is not used for motion prediction of the CU / PU immediately following the current block in the processing order.

[0611] 3.12. The method as described in any one of Examples 3.1-3.11, wherein the updated motion vector is used only as a predictor for processing subsequent images / pieces / strips.

[0612] 3.13. The method as described in Example 3.12, wherein the updated motion vector is used as temporal motion vector prediction (TMVP) in Advanced Motion Vector Prediction (AMVP) mode, Merge mode, or affine mode.

[0613] 3.14. The method as described in Example 3.12, wherein the updated motion vector is used only as a predictor for processing subsequent images / pieces / strips in Optional Time Motion Vector Prediction (ATMVP) mode and / or Space-Time Motion Vector Prediction (STMVP) mode.

[0614] 3.15. The method as described in any one of Examples 3.1-3.14, wherein the encoder signals to the decoder information including whether to update the MV for the BIO coded block and / or whether to use the updated MV for motion prediction and / or how to use the updated MV for motion prediction.

[0615] 3.16. The method as described in Example 3.15 further includes: signaling the information in a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), strip header, coding tree unit (CTU), or CU.

[0616] 3.17. The method as described in Example 3.1 further includes: updating motion information, which includes updating the motion vector and reference image for each predicted direction at the block level.

[0617] 3.18. The method as described in Example 3.1 or 3.17, wherein, within a block, for some sub-blocks, the updated motion information is stored, and for the remaining sub-blocks, the unupdated motion information is stored.

[0618] 3.19. As described in Example 3.1 or 3.17, the updated motion vector is stored only for internal sub-blocks not located at the PU / CU / CTU boundary.

[0619] 3.20. The method as described in Example 3.19 further includes: using the updated motion vector of the inner sub-block for motion prediction, deblocking, or OBMC.

[0620] 3.21. The method described in Example 3.1 or 3.17 stores the updated motion vector only for the boundary sub-blocks at the PU / CU / CTU boundary.

[0621] 3.22. The method as described in Example 3.1 or 3.17, wherein if the adjacent block and the current block are not in the same CTU or in the same region having a size of 64×64 or 32×32, then the updated motion information from the adjacent block is not used.

[0622] 3.23. The method as described in Example 3.22, wherein if the adjacent block and the current block are not in the same CTU or in the same region having a size of 64×64 or 32×32, the adjacent block is marked as unavailable.

[0623] 3.24. The method as described in Example 3.22, wherein if the adjacent block and the current block are not in the same CTU or in the same region having a size of 64×64 or 32×32, the current block uses unupdated motion information.

[0624] 3.25. The method as described in Example 3.17, wherein if the adjacent block and the current block are not in the same CTU row or in the same row of a region having a size of 64×64 or 32×32, then the updated motion vector from the adjacent block is not used.

[0625] 3.26. The method as described in Example 3.25, wherein if the adjacent block and the current block are not in the same CTU row or in the same row of a region having a size of 64×64 or 32×32, the adjacent block is marked as unavailable.

[0626] 3.27. The method as described in Example 3.25, wherein if the adjacent block and the current block are not in the same CTU row or in the same row of a region having a size of 64×64 or 32×32, the current block uses the unupdated motion information from the adjacent block.

[0627] 3.28. The method as described in Example 3.17, wherein if the bottom row of the block is a CTU or the bottom row of a region having a size of 64×64 or 32×32, the motion information of the block is not updated.

[0628] 3.29. The method as described in Example 3.17, wherein if the rightmost column of the block is a CTU or the rightmost column of a region having a size of 64×64 or 32×32, the motion information of the block is not updated.

[0629] 3.30. The method as described in Example 3.1 or 3.17 further includes predicting the motion information of blocks / CUs within the current CTU based on updated or unupdated motion information of neighboring CTUs or regions.

[0630] 3.31. The method as described in Example 3.30, wherein the updated motion information from the left CTU or the left region is used for the current CTU.

[0631] 3.32. The method as described in Example 3.30 or 3.31, wherein the updated motion information from the upper left CTU or the upper left region is used for the current CTU.

[0632] 3.33. The method as described in any one of Examples 3.30-3.32, wherein the updated motion information from the upper CTU or the upper region is used for the current CTU.

[0633] 3.34. The method as described in any one of Examples 3.30-3.33, wherein the updated motion information from the upper right CTU or the upper right region is used for the current CTU.

[0634] 3.35. The method as described in any one of Examples 3.30-3.34, wherein each of one or more regions has a size of 64×64 or 32×32.

[0635] 3.36. The method as described in any one of Examples 3.1-3.35, wherein the method is not applied if the current block satisfies a specific condition.

[0636] 3.37. The method as described in Example 3.36, wherein the specific condition specifies at least one of the following: the size of the current block, the strip type of the current block, the picture type of the current block, and the slice type of the current block.

[0637] 3.38. The method as described in Example 3.36, wherein the specific condition specifies that the number of samples contained in the current block is less than a first threshold.

[0638] 3.39. The method as described in Example 3.36, wherein the specific condition specifies that the minimum dimensions of the width and height of the current block are less than or not greater than a second threshold.

[0639] 3.40. The method as described in Example 3.36, wherein the specific condition specifies that the width of the current block is less than or not greater than a third threshold, and / or the height of the current block is less than or not greater than a fourth threshold.

[0640] 3.41. The method as described in Example 3.36, wherein the specific condition specifies that the width of the current block is greater than or not less than a third threshold, and / or the height of the current block is greater than or not less than a fourth threshold.

[0641] 3.42. The method as described in Example 3.36, wherein the method is applied to the sub-block level if the width and / or height of the block to which the sub-block belongs is equal to or greater than a fifth threshold.

[0642] 3.43. The method as described in Example 3.42, wherein the current block is divided into a plurality of sub-blocks, and each of the plurality of sub-blocks is further subjected to bidirectional optical flow (BIO) processing in the same manner as a normal coded block having a size equal to that of the sub-block.

[0643] 3.44. The method as described in any one of Examples 3.38-3.42, wherein each of the first to fifth thresholds is predefined or signaled at the Sequence Parameter Set (SPS) level, or the Picture Parameter Set (PPS) level, or the picture level, or the strip level or the slice level.

[0644] 3.45. The method as described in Example 3.44, wherein each of the first to fifth thresholds is defined based on encoded information including at least one of block size, image type, and temporal layer index.

[0645] 3.46. An apparatus in a video system, comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any one of Examples 3.1 to 3.45.

[0646] 3.47. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method of any one of Examples 3.1 to 3.45.

[0647] 4.1. A video processing method, comprising:

[0648] Determine the MVD precision for the current block to be processed in affine mode from the motion vector difference (MVD) precision set;

[0649] Based on the determined MVD precision, a conversion is performed between the current block and the bitstream representation of the video including the current block.

[0650] 4.2. The method as described in Example 4.1, wherein the MVD represents the difference between the predicted motion vector and the actual motion vector used during motion compensation processing.

[0651] 4.3. The method as described in Example 4.2, wherein the MVD precision set comprises a plurality of different MVD precisions constituting a geometric sequence.

[0652] 4.4. The method as described in Example 4.3, wherein the MVD precision set includes 1 / 4, 1, and 4 pixel MVD precisions.

[0653] 4.5. The method as described in Example 4.3, wherein the MVD precision set includes 1 / 4, 1 / 2, 1, 2 and 4 pixel MVD precisions.

[0654] 4.6. The method as described in Example 4.3, wherein the MVD precision set includes 1 / 16, 1 / 8 and 1 / 4 pixel MVD precision.

[0655] 4.7. The method as described in Example 4.1, wherein the current block is a coding unit or a prediction unit.

[0656] 4.8. The method as described in any one of Examples 4.1-4.7, wherein determining the MVD accuracy further includes:

[0657] The MVD precision of the current block is determined based on the syntax element that indicates the MVD precision.

[0658] 4.9. The method as described in Example 4.8, wherein the syntax element exists when a non-zero MVD component of the current block exists.

[0659] 4.10. The method as described in Example 4.8, wherein the syntax element does not exist when there is no non-zero MVD component of the current block.

[0660] 4.11. The method as described in Example 4.8, wherein the syntax element exists regardless of whether any non-zero MVD component of the current block exists.

[0661] 4.12. The method as described in Example 4.1, wherein the current block is processed using an affine inter-frame mode or an affine advanced motion vector prediction (AMVP) mode.

[0662] 4.13. The method as described in Example 4.12, wherein different MVDs of the current block are associated with different MVD accuracies.

[0663] 4.14. The method as described in Example 4.13, wherein the affine inter-frame mode is a 4-parameter affine inter-frame mode with 2 control points, and an MVD is used for each control point in each prediction direction.

[0664] 4.15. The method as described in Example 4.14, wherein the two control points are associated with different MVD accuracies.

[0665] 4.16. The method as described in Example 4.13, wherein the affine inter-frame mode is a 6-parameter affine inter-frame mode with 3 control points, and an MVD is used for each control point in each prediction direction.

[0666] 4.17. The method as described in Example 4.16, wherein the three control points are associated with different MVD accuracies.

[0667] 4.18. The method as described in Example 4.15, wherein there are two syntax elements to indicate the different MVD precisions of the two control points.

[0668] 4.19. The method as described in Example 4.17, wherein there are three syntax elements to indicate different MVD accuracies for the three control points.

[0669] 4.20. The method as described in Example 4.1, wherein the MVD precision set is determined based on the encoding information of the current block.

[0670] 4.21. The method as described in Example 4.20, wherein the encoding information includes the quantization level of the current block.

[0671] 4.22. The method as described in Example 4.21, wherein a coarser MVD precision set is selected for larger quantization levels.

[0672] 4.23. The method as described in Example 4.21, wherein a finer set of MVD precision is selected for smaller quantization levels.

[0673] 4.24. The method as described in any one of Examples 4.1-4.23, wherein the method is not applied if the current block satisfies a specific condition.

[0674] 4.25. The method as described in Example 4.24, wherein the specific condition specifies at least one of the following: the size of the current block, the strip type of the current block, the picture type of the current block, and the slice type of the current block.

[0675] 4.26. The method as described in Example 4.24, wherein the specific condition specifies that the number of samples contained in the current block is less than a first threshold.

[0676] 4.27. The method as described in Example 4.24, wherein the specific condition specifies that the minimum dimensions of the width and height of the current block are less than or not greater than a second threshold.

[0677] 4.28. The method as described in Example 4.24, wherein the specific condition specifies that the width of the current block is less than or not greater than a third threshold, and / or the height of the current block is less than or not greater than a fourth threshold.

[0678] 4.29. The method as described in Example 4.24, wherein the specific condition specifies that the width of the current block is greater than or not less than a third threshold, and / or the height of the current block is greater than or not less than a fourth threshold.

[0679] 4.30. The method as described in Example 4.24, wherein the method is applied at the sub-block level if the width and / or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

[0680] 4.31. The method as described in Example 4.30, wherein the current block is divided into a plurality of sub-blocks, and each of the plurality of sub-blocks further undergoes a bidirectional optical flow (BIO) process in the same manner as a normal coded block having a size equal to the size of the sub-block.

[0681] 4.32. The method as described in any one of Examples 4.26-4.30, wherein each of the first to fifth thresholds is predefined or signaled at the Sequence Parameter Set (SPS) level, or the Picture Parameter Set (PPS) level, or the picture level, or the strip level or the slice level.

[0682] 4.33. The method as described in Example 4.32, wherein each of the first to fifth thresholds is defined based on encoding information including at least one of block size, image type, and temporal layer index.

[0683] 4.34. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any one of Examples 4.1 to 4.33.

[0684] 4.35. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the methods of any one of Examples 4.1 to 4.33.

[0685] 5.1. A video processing method, comprising:

[0686] Determine the outdated motion information associated with the current block;

[0687] The outdated motion information is updated based on a multi-decoder-side motion vector derivation (DMVD) method to generate updated motion information for the current block; and

[0688] Based on the updated motion information, a conversion is performed between the current block and the bitstream representation of the video including the current block.

[0689] 5.2. The method as described in Example 5.1, wherein the plurality of DMVD methods include at least two of the following: bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate upconversion (FRUC) technique, and template matching technique.

[0690] 5.3. The method as described in Example 5.2, wherein the plurality of DMVD methods are executed simultaneously on the unupdated motion information of the current block, and the unupdated motion vector of the unupdated motion information is input as the search starting point of each of the plurality of DMVD methods.

[0691] 5.4. The method as described in Example 5.2, wherein the plurality of DMVD methods are executed in a cascade manner on the unupdated motion information of the current block, and the updated motion vector of the updated motion information generated by one DMVD method is used as the search starting point for the next DMVD method.

[0692] 5.5. The method as described in Example 5.4, wherein the first DMVD method is DMVR, and the next DMVD method is BIO, wherein DMVR is performed on the unupdated motion information of the current block to generate the updated motion information, and the updated motion vector of the updated motion information is used as the search starting point for BIO.

[0693] 5.6. The method as described in any one of Examples 5.1 to 5.5, wherein updating the unupdated motion information based on the multiple decoder-side motion vector derivation (DMVD) method to generate updated motion information for the current block further includes:

[0694] Multiple sets of updated motion information are derived using the aforementioned DMVD methods.

[0695] The final set of updated motion information is generated from the multiple sets of updated motion information.

[0696] 5.7. The method as described in Example 5.6, wherein generating the final set of updated motion information from the plurality of sets of updated motion information further includes:

[0697] The final set of updated motion information is generated based on the average or weighted average of the multiple sets of updated motion information.

[0698] 5.8. The method as described in Example 5.6, wherein generating the final set of updated motion information from the plurality of sets of updated motion information further includes:

[0699] The final set of updated motion information is generated by filtering the multiple sets of updated motion information using a median filter.

[0700] 5.9. The method as described in Example 5.6, wherein generating the final set of updated motion information from the plurality of sets of updated motion information further includes:

[0701] Assign different priorities to the multiple DMVD methods.

[0702] The set of updated motion information derived by the DMVD method with the highest priority is selected as the updated motion information of the final set.

[0703] 5.10. The method as described in Example 5.9, wherein the highest priority is assigned to the decoder-side motion vector refinement (DMVR).

[0704] 5.11. The method as described in any one of Examples 5.1 to 5.5, wherein performing the conversion between the current block and a bitstream representation of the video including the current block based on the updated motion information further comprises:

[0705] Motion compensation is performed using multiple sets of updated motion information derived from the aforementioned multiple DMVD methods to obtain multiple sets of motion compensation results.

[0706] The current block is generated based on the average or weighted average of the multiple sets of motion compensation results.

[0707] 5.12. The method as described in any one of Examples 5.1 to 5.5, wherein performing the conversion between the current block and a bitstream representation of the video including the current block based on the updated motion information further comprises:

[0708] Motion compensation is performed using multiple sets of updated motion information derived from the aforementioned multiple DMVD methods to obtain multiple sets of motion compensation results.

[0709] The current block is generated by filtering the multiple sets of motion compensation results using a median filter.

[0710] 5.13. The method as described in any one of Examples 5.1 to 5.5, wherein updating the unupdated motion information based on the multiple decoder-side motion vector derivation (DMVD) method to generate updated motion information for the current block further includes:

[0711] Assign different priorities to the multiple DMVD methods.

[0712] Select the DMVD method with the highest N priorities and that is valid from the plurality of DMVD methods, where N is an integer and N>=1.

[0713] Updated motion information is generated for the current block based on N DMVD methods.

[0714] 5.14. The method as described in any one of Examples 5.1 to 5.13, wherein the current block is a prediction unit.

[0715] 5.15. The method as described in any one of Examples 5.1 to 5.14, wherein the unupdated motion information comprises an unupdated motion vector and a reference image for each predicted direction.

[0716] 5.16. The method as described in any one of Examples 5.1-5.15, wherein the method is not applied if the current block satisfies a specific condition.

[0717] 5.17. The method as described in Example 5.16, wherein the specific condition specifies at least one of the following: the size of the current block, the strip type of the current block, the picture type of the current block, and the slice type of the current block.

[0718] 5.18. The method as described in Example 5.16, wherein the specific condition specifies that the number of samples contained in the current block is less than a first threshold.

[0719] 5.19. The method as described in Example 5.16, wherein the specific condition specifies that the minimum dimensions of the width and height of the current block are less than or not greater than a second threshold.

[0720] 5.20. The method as described in Example 5.16, wherein the specific condition specifies that the width of the current block is less than or not greater than a third threshold, and / or the height of the current block is less than or not greater than a fourth threshold.

[0721] 5.21. The method as described in Example 5.16, wherein the specific condition specifies that the width of the current block is greater than or not less than a third threshold, and / or the height of the current block is greater than or not less than a fourth threshold.

[0722] 5.22. The method as described in Example 5.16, wherein the method is applied at the sub-block level if the width and / or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

[0723] 5.23. The method as described in Example 5.22, wherein the current block is divided into a plurality of sub-blocks, and each of the plurality of sub-blocks further undergoes a bidirectional optical flow (BIO) process in the same manner as a normal coded block having a size equal to that of the sub-block.

[0724] 5.24. The method as described in any one of Examples 5.18-5.22, wherein each of the first to fifth thresholds is predefined or signaled at the Sequence Parameter Set (SPS) level, or the Picture Parameter Set (PPS) level, or the picture level, or the strip level or the slice level.

[0725] 5.25. The method as described in Example 5.24, wherein each of the first to fifth thresholds is defined based on encoded information including at least one of block size, image type, and temporal layer index.

[0726] 5.26. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any one of Examples 5.1 to 5.25.

[0727] 5.27. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the methods of any one of Examples 5.1 to 5.25.

[0728] As can be understood from the foregoing, specific embodiments of the disclosed technology have been described herein for illustrative purposes, but various modifications can be made without departing from the scope of the invention. Therefore, the disclosed technology is not limited except for the appended claims.

[0729] The implementation and functional operation of the subject matter described in this patent document can be implemented in various systems, digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations thereof. The subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more transient and non-transitory computer program instruction modules encoded on a computer-readable medium for operation by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of material affecting machine-readable propagation signals, or a combination thereof. The terms "data processing unit" and "data processing apparatus" include all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may include code that creates an operating environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof.

[0730] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on one or more computers located at a single site or distributed across multiple sites interconnected by a communication network.

[0731] The processing and logic flows described in this specification can be executed by one or more programmable processors that run one or more computer programs to perform functions by manipulating input data and generating output. The processing and logic flows can also be executed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0732] For example, processors suitable for running computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor that executes instructions and one or more storage devices that store those instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to one or more mass storage devices to receive data from or transfer data to, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices. Processors and memory may be supplemented by or incorporated into special-purpose logic circuitry.

[0733] Intended to include the instruction manual and accompanying Figure 1 The above is taken as exemplary only, where exemplary means example. As used herein, the singular forms “a,” “one,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Additionally, unless the context clearly indicates otherwise, the use of “or” is intended to include “and / or.”

[0734] While this patent document contains numerous details, it should not be construed as limiting the scope of any invention or claim, but rather as a description of features specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various functions described in the context of a single embodiment may also be implemented individually in multiple embodiments, or in any suitable sub-combination. Furthermore, while features may be described as functioning in certain combinations, or even initially claimed in this way, in some cases one or more features may be removed from the claimed combination, and the claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0735] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as meaning that these operations must be performed in the specific order or sequence shown, or that all of the operations shown must be performed, in order to obtain the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0736] Only some implementation methods and examples have been described. Other implementation methods, enhancements and variations can be made based on the content described and illustrated in this patent document.

Claims

1. A video processing method, comprising: Determine the outdated motion information associated with the current block; The unupdated motion information is updated based on the multi-decoder-side motion vector derivation (DMVD) method to generate updated motion information for the current block; as well as Based on the updated motion information, a conversion is performed between the current block and the bitstream of the video including the current block. The plurality of DMVD methods mentioned therein include at least two of the following: bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate upconversion (FRUC) technique, and template matching technique, and Specifically, the multiple DMVD methods are executed on the unupdated motion information of the current block in a cascading manner, and the updated motion vector of the updated motion information generated by one DMVD method is used as the search starting point for the next DMVD method.

2. The method as described in claim 1, wherein, The plurality of DMVD methods are executed simultaneously on the unupdated motion information of the current block, and the unupdated motion vector of the unupdated motion information is used as the search starting point for each of the plurality of DMVD methods.

3. The method as described in claim 1, wherein, The first DMVD method is DMVR, and the next DMVD method is BIO, wherein DMVR is performed on the unupdated motion information of the current block to generate the updated motion information, and the updated motion vector of the updated motion information is used as the search starting point for BIO.

4. The method according to any one of claims 1 to 3, wherein, The step of performing the conversion between the current block and the bitstream of the video including the current block based on the updated motion information further includes: Motion compensation is performed using multiple sets of updated motion information derived from the aforementioned multiple DMVD methods to obtain multiple sets of motion compensation results. The current block is generated based on the average or weighted average of the multiple sets of motion compensation results.

5. The method according to any one of claims 1 to 3, wherein, The step of performing the conversion between the current block and the bitstream of the video including the current block based on the updated motion information further includes: Motion compensation is performed using multiple sets of updated motion information derived from the aforementioned multiple DMVD methods to obtain multiple sets of motion compensation results. The current block is generated by filtering the multiple sets of motion compensation results using a median filter.

6. The method according to any one of claims 1 to 3, wherein, The method of updating the unupdated motion information based on multiple decoder-side motion vector derivation (DMVD) to generate updated motion information for the current block further includes: Assign different priorities to the multiple DMVD methods. Select the DMVD method with the highest N priorities and that is valid from the plurality of DMVD methods, where N is an integer and N>= 1. Updated motion information is generated for the current block based on N DMVD methods.

7. The method according to any one of claims 1 to 3, wherein, The current block is a prediction unit.

8. The method according to any one of claims 1 to 3, wherein, The unupdated motion information includes unupdated motion vectors and reference images for each predicted direction.

9. The method of claim 1, wherein, The method is not applied if the current block meets certain conditions.

10. The method of claim 9, wherein, The specific condition specifies at least one of the following: the size of the current block, the strip type of the current block, the image type of the current block, and the slice type of the current block.

11. The method of claim 9, wherein, The specific condition specifies that the number of samples contained in the current block is less than a first threshold.

12. The method of claim 9, wherein, The specific condition specifies that the minimum dimensions of the width and height of the current block are less than or not greater than a second threshold.

13. The method of claim 9, wherein, The specific condition specifies that the width of the current block is less than or no greater than a third threshold, and / or the height of the current block is less than or no greater than a fourth threshold.

14. The method of claim 9, wherein, The specific condition specifies that the width of the current block is greater than or not less than a third threshold, and / or the height of the current block is greater than or not less than a fourth threshold.

15. The method of claim 9, wherein, The method is applied at the sub-block level if the width and / or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

16. The method of claim 15, wherein, The current block is divided into multiple sub-blocks, and each of the multiple sub-blocks further undergoes a bidirectional optical flow (BIO) process in the same manner as a normal coded block having a size equal to that of the sub-block.

17. The method according to any one of claims 11-15, wherein, Each threshold is predefined at the Sequence Parameter Set (SPS) level, or at the Picture Parameter Set (PPS) level, or at the picture level, or at the strip level or slice level, or is signaled.

18. The method of claim 17, wherein, Each threshold is defined based on encoded information including at least one of the following: block size, image type, and temporal layer index.

19. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 18.

20. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 18.

Images