Decoding method of coded video bitstream, video decoder and storage medium

Abstract

A decoding method, video decoder, and storage medium for encoded video bitstreams are provided, wherein the method is used to decode video bitstreams encoded in a Temporal Interpolation Prediction (TIP) mode. First and second motion vectors pointing to corresponding reference frames or reference images within these frames are generated for blocks of the current image. Then, based on a bidirectional matching process, the aforementioned motion vectors are improved by applying a decoder-side motion vector improvement (DMVR) process, and the blocks are decoded using the improved motion vectors. This improvement may more specifically involve considering candidate improved motion vectors selected by bidirectional matching. This improvement can be applied to block and sub-block partitions of the current image.

Description

[0001] Related patents and applications

[0002] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 345,329, filed May 24, 2022, and U.S. Patent Application No. 17 / 982,071, filed November 7, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to video encoding and decoding, and more specifically, to methods and apparatus for improving the accuracy of sports field prediction by using temporal interpolation. Background Technology

[0004] With the widespread growth of connected devices and digital media, video encoding and decoding are widely used. AOMedia Video1 (AV1) is an open video coding format designed specifically for video transmission over the Internet. Many components of the AV1 project are derived from previous research. While AV1 is an improvement on existing solutions (e.g., its predecessor, the codec VP9), interpolation issues remain. Therefore, further improvements are needed. Summary of the Invention

[0005] According to certain embodiments of this disclosure, a method for decoding an encoded video bitstream is provided. The method is performed by at least one processor in a video decoder. The method includes receiving an encoded video bitstream comprising a current image, the current image including at least one block and a syntax element indicating that at least one block should be predicted in a temporal interpolated prediction (TIP) mode. The method further includes generating a motion field for the at least one block, the motion field including a first motion vector pointing to a first reference image and a second motion vector pointing to a second reference image. The method also includes generating a first refined motion vector and a second refined motion vector based on a bidirectional matching process, by using the first and second motion vectors during a decoder-side motion vector refinement (DMVR) process for the at least one block. The method further includes decoding the at least one block using the first refined motion vector and the second refined motion vector.

[0006] According to other embodiments of this disclosure, a video decoder is provided. The video decoder includes: at least one communication module configured to receive a bitstream; at least one non-volatile memory configured to store computer program code; and at least one processor operatively connected to the at least one communication module and the at least one non-volatile memory. The at least one processor is configured to operate according to instructions of the computer program code. The computer program code includes receiving code configured to cause at least one of the at least one processor to receive, via the at least one communication module, an encoded video bitstream including a current image, the current image including at least one block and syntax elements, the syntax elements indicating at least one block to be predicted in a temporal interpolation prediction (TIP) mode. The computer program code also includes generation code configured to cause at least one of the at least one processor to generate a motion field for at least one block, the motion field including a first motion vector pointing to a first reference image and a second motion vector pointing to a second reference image. The computer program code also includes enhancement code configured to cause at least one of the at least one processor to generate a first enhanced motion vector and a second enhanced motion vector based on a bidirectional matching process by using a first motion vector and a second motion vector during a decoder-side motion vector enhancement (DMVR) process for at least one block. The computer program code also includes decoding code configured to cause at least one of the at least one processor to decode at least one block using the first enhanced motion vector and the second enhanced motion vector.

[0007] According to other embodiments of this disclosure, a non-transitory computer-readable recording medium is provided. Instructions executable by at least one processor are recorded on the recording medium to perform a method for decoding an encoded video bitstream. The method includes receiving an encoded video bitstream including a current picture, the current picture including at least one block and syntax elements, the syntax elements indicating that at least one block should be predicted in a temporal interpolation prediction (TIP) mode. The method further includes generating a motion field for the at least one block, the motion field including a first motion vector pointing to a first reference picture and a second motion vector pointing to a second reference picture. The method further includes generating a first improved motion vector and a second improved motion vector based on a bidirectional matching process by using the first and second motion vectors in a decoder-side motion vector improvement (DMVR) process for the at least one block. The method further includes decoding the at least one block using the first and second improved motion vectors.

[0008] Other aspects will be set forth in part in the description below, and some will be obvious from the description or may be realized by practicing the embodiments presented in this disclosure. Attached Figure Description

[0009] Features, aspects and advantages of certain exemplary embodiments of the present disclosure will now be described with reference to the accompanying drawings, in which like reference numerals denote like elements, wherein:

[0010] Figure 1 An illustrative example of AV1 tree partitioning according to an exemplary embodiment is depicted;

[0011] Figure 2 An illustrative example of partitioning blocks using a quadtree plus binary tree structure according to an exemplary embodiment is depicted;

[0012] Figure 3 An illustrative example of a block partitioning structure using a ternary tree according to an exemplary embodiment is depicted;

[0013] Figure 4 An illustrative operation of a derived spatial motion vector predictor according to an exemplary embodiment is described;

[0014] Figure 5 An illustrative operation of a time motion vector predictor according to an exemplary embodiment is described;

[0015] Figure 6 A set of illustrative predefined block locations for deriving a time motion predictor, according to an exemplary embodiment, are depicted;

[0016] Figure 7 An illustrative operation for generating motion vector candidates via a single inter-frame prediction block, according to an exemplary embodiment, is described;

[0017] Figure 8 An illustrative operation for generating motion vector candidates via composite inter-frame prediction blocks according to an exemplary embodiment is described;

[0018] Figure 9 An illustrative operation for updating the motion vector candidate library according to an exemplary embodiment is described;

[0019] Figure 10 This is a flowchart depicting the process of constructing a list of motion vector predictions according to an exemplary embodiment;

[0020] Figure 11 An illustrative operation of a composite inter-frame prediction mode according to an exemplary embodiment is described;

[0021] Figure 12 An illustrative operation of a time interpolation prediction mode according to an exemplary embodiment is described;

[0022] Figure 13 An illustrative operation based on bidirectional matching and improved decoder-side motion vectors according to an exemplary embodiment is described;

[0023] Figure 14 An illustrative use case of a merging mode with motion vector difference according to an exemplary embodiment is depicted; and

[0024] Figure 15 The diagram shows an example component of a device on which embodiments of the systems and / or methods described herein may be implemented. Detailed Implementation

[0025] The following detailed description of exemplary embodiments is with reference to the accompanying drawings. The same reference numerals in different drawings may identify the same or similar elements.

[0026] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible based on the foregoing disclosure, or may be obtained from practice of the implementations. Furthermore, one or more features or components of one embodiment may be incorporated into or combined with another embodiment (or one or more features of another embodiment). Additionally, in the flowcharts and operational descriptions provided below, it should be understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least partially), and the order of one or more operations may be switched.

[0027] Clearly, the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit these implementations. Therefore, the operation and behavior of the systems and / or methods described herein do not refer to any specific software code. It should be understood that software and hardware can be designed to implement the systems and / or methods described herein.

[0028] Even if specific combinations of features are stated in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically stated in the claims and / or not disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes combinations of each dependent claim with each other claim in the claim set.

[0029] Unless explicitly stated otherwise, no element, action, or instruction used herein should be construed as critical or necessary. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” If only one item is involved, the term “an” or similar wording is used. Additionally, as used herein, the terms “possess,” “have,” “have,” “include,” “contain,” etc., are intended to indicate open-ended terms. Furthermore, the phrase “based on” is intended to mean “at least partially based on” unless explicitly stated otherwise. Furthermore, expressions such as “at least one of [A] and [B]” or “at least one of [A] or [B]” should be understood to include only A, only B, or both A and B.

[0030] With the current surge in media access via the internet, video encoding has become increasingly important for reducing network load. Methods and apparatus for video encoding are disclosed.

[0031] Figure 1 An illustrative example of an AV1 partition tree 100 according to an exemplary embodiment is depicted. In the partition tree 100 of image 110, a portion 115 of image 110 (referred to as a superblock in VP9 / AV1 terminology) is expanded into a ten-way structure 120, which partitions the superblock 115 according to various partitioning patterns (e.g., 125a, 125b, 125c) that can all be processed. While partitioning patterns using rectangular partitions cannot be further subdivided, partitioning pattern 125c consists only of square patterns, which themselves can be partitioned in the same way as the superblock 115, resulting in recursive partitioning.

[0032] The partitions or blocks in this process can also be called coding tree units (CTUs), and a group of pixels or pixel data units represented by CTUs can be called coding tree blocks (CTBs). Note that a single CTU can represent multiple CTBs, where each CTB represents a different information component (e.g., a CTB represents luminance information, and multiple CTBs represent different color components, such as "red," "green," and "blue" factors).

[0033] Compared to the 64×64 pixel superblock in VP9, ​​AV1 increases the maximum possible size of the starting superblock 115 to, for example, 128×128 pixels. Furthermore, the ten-way architecture 120 includes 4:1 and 1:4 rectangular partitioning patterns 125a and 125b, which were not present in VP9. In addition, AV1 adds more flexibility for the use of partitions below the 8×8 pixel level; in a sense, 2×2 chroma inter-frame prediction is now possible in certain situations.

[0034] In High Efficiency Video Coding (HEVC), coding tree units can be divided into coding units (CUs) using a quadtree structure represented as a coding tree to accommodate various local characteristics. At the CU level, it can be determined whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode picture regions. Depending on the PU partitioning type, each CU can be further divided into one, two, or four prediction units (PUs). Within a PU, the same prediction process can be applied, and relevant information can be transferred to the decoder based on the PU. After obtaining the residual block by applying the prediction process based on the PU partitioning type, the CU can be partitioned into transform units (TUs) according to another quadtree structure (e.g., the coding tree of the CU). The HEVC structure has multiple partitioning concepts, including CUs, PUs, and TUs. In HEVC, for inter-frame prediction blocks, CUs or TUs can be square, while PUs can be square or rectangular. In HEVC, a coding block can be further divided into four square sub-blocks, and a transform can be performed on each sub-block (i.e., TU). Each TU can be further recursively divided (using quadtree partitioning) into smaller TUs, a process known as residual quadtree (RQT). At image boundaries, HEVC can employ implicit quadtree partitioning, allowing blocks to maintain quadtree partitioning until their size fits the image boundaries.

[0035] Figure 2 An illustrative example of partitioning a CTU 220 using a quadtree plus binary tree (QTBT) structure 210 according to an exemplary embodiment is depicted. The QTBT structure 210 includes quadtree nodes and binary tree nodes. Figure 2 In the diagram, solid lines represent branches and leaves, as well as corresponding block partitions resulting from partitions at quadtree nodes (e.g., node 211a), and dashed lines represent branches and leaves, as well as corresponding block partitions resulting from partitions at binary tree nodes (e.g., node 211b).

[0036] A split at a binary tree node divides the corresponding block into two equal-sized sub-blocks. For each split (i.e., non-leaf) binary tree node (e.g., node 211b), a flag or other marker can be used to indicate which split type (i.e., horizontal or vertical) is used, where, for example, 0 indicates a horizontal split and 1 indicates a vertical split. A split at a quadtree node (e.g., node 211a) divides the corresponding block horizontally and vertically into four equal-sized sub-blocks, so the flag indicating the split type can be omitted.

[0037] Furthermore, the QTBT scheme supports the flexibility of having independent QTBT structures for luma and chroma. For P and B slices, the luma and chroma CTBs within a single CTU can share the same QTBT structure. However, for I slices, the luma CTB can be partitioned into CUs by a QTBT structure, while the chroma CTB can be partitioned into chroma CUs by different QTBT structures. This means that a CU in an I slice can include a coding block for the luma component or coding blocks for both chroma components, while a CU in a P or B slice can include coding blocks for all three color components.

[0038] In HEVC, inter-frame prediction for small blocks is restricted to reduce memory accesses for motion compensation, making bidirectional prediction unsupported for 4×8 and 8×4 blocks, and inter-frame prediction unsupported for 4×4 blocks. In QTBT implementations in some embodiments, these restrictions are removed.

[0039] In HEVC, the CTU can be divided into CUs using a quadtree represented as a coding tree to accommodate various local characteristics. At the CU level, it can be determined whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode picture regions. Depending on the PU partitioning type, each CU can be further divided into one, two, or four PUs. Within a PU, the same prediction process can be applied, and relevant information can be transferred to the decoder based on the PU. After obtaining residual blocks by applying the prediction process based on the PU partitioning type, the CU can be partitioned into transform units (TUs) according to another quadtree structure, just like the coding tree of the CU. Therefore, the HEVC structure can have multiple partitioning concepts, including CUs, PUs, and TUs.

[0040] according to Figure 2 The illustrated embodiment of QTBT structure 210 eliminates the concept of multiple partition types, namely, the separation of CU, PU, ​​and TU concepts, and supports greater flexibility in the shape of CU partitions. In the QTBT block structure, CUs can have square or rectangular shapes. For example... Figure 2 As shown, the coding tree unit (CTU) 220 can first be partitioned based on the quadtree node 211a of the QTBT structure 210. The branches of the quadtree node 211a can be further partitioned based on binary tree nodes (e.g., nodes 211b and 211c) or another quadtree node (e.g., node 211d). There are two possible partitioning types in the binary tree partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. The leaf nodes of the binary tree can be designated as coding units (CUs), and this partitioning can be used for prediction and transform processing without any further partitioning. This illustrates that in the QTBT coding block structure, CUs, PUs, and TUs can have the same block size.

[0041] In some embodiments, a CU may include coded blocks (CBs) of different color components (e.g., in the case of P and B slices in a 4:2:0 chroma format, a CU may contain one luma CB and two chroma CBs) or alternatively include a CB of a single component (e.g., in the case of an I slice, a CU may contain one luma CB or two chroma CBs).

[0042] Define the following parameters for the QTBT partitioning scheme.

[0043] –CTU size: The size of the root node of the quadtree, the same concept as in HEVC.

[0044] –MinQTSize: The minimum allowed size of a quadtree leaf node

[0045] –MaxBTSize: The maximum allowed size of the root node of the binary tree.

[0046] –MaxBTDepth: Maximum allowed binary tree depth

[0047] –MinBTSize: The minimum allowed size of a binary leaf node

[0048] In the example implementation of the QTBT segmentation structure, the size of the CTU 220 can be set to 128×128 luminance samples with two corresponding 64×64 chrominance sample blocks. The MinQTSize is set to 16×16, the MaxBTSize is set to 64×64, the MinBTSize (for width and height) is set to 4×4, and the MaxBTDepth is set to 4.

[0049] In this implementation, quadtree partitioning is applied to the CTU 220 represented by the quadtree root node 211a to generate quadtree leaf nodes 211b, 211c, 211d, and 211e. Quadtree leaf nodes 211b, 211c, 211d, and 211e can have sizes ranging from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If a leaf quadtree node is 128×128 in size, it will not be further partitioned by the binary tree because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, the leaf quadtree node can be further partitioned by the QTBT partitioning structure 210. Therefore, quadtree leaf node 211b can also be considered as the root node of a binary tree with a depth of 0.

[0050] When the depth of the binary tree reaches MaxBTDepth (i.e., 4), further splitting is no longer considered. When the width of a binary tree node equals MinBTSize (i.e., 4), further horizontal splitting is not considered. Similarly, when the height of a binary tree node equals MinBTSize, further vertical splitting is not considered.

[0051] Once segmentation stops, the final leaf nodes (e.g., leaf node 211f) of the QTBT partition structure 210 can be further processed through prediction and transformation. In some embodiments, the maximum CTU size is 256×256 luminance samples.

[0052] Figure 3 An illustrative example of a block partitioning structure using a ternary tree, such as a VVC Multi-Type Tree (MTT) structure, is depicted according to exemplary embodiments. In addition to partitioning implemented by the QTBT partitioning structure described above, the use of a ternary tree is added to the partitioning structure, with its markers or labels similar to those used in binary tree nodes, enabling center-side ternary tree partitioning with a vertical 310° and a horizontal 320° orientation. Ternary tree partitioning complements quadtree and binary tree partitioning: ternary tree partitioning can capture objects located at the center of blocks divided by quadtree or binary tree partitioning. The width and height of a ternary tree partition can both be powers of 2, eliminating the need for additional transformations.

[0053] In theory, the complexity of traversing a tree is T^D, where T represents the number of partition types and D is the depth of the tree. Therefore, to reduce complexity, the tree can be a two-level tree (D = 2).

[0054] Figure 4 An illustrative operation of a spatial motion vector predictor (SVMP) according to an exemplary embodiment is depicted. The spatial motion vector predictor (SVMP) itself may take the form of motion vectors or include motion vectors. The SVMP can be derived from blocks adjacent to the current block 410. More specifically, the SVMP can be derived from spatially adjacent blocks 420 that are adjacent or close to the current block 410 on the top and left. For example, in Figure 4 In this context, a block is spatially adjacent to block 420 if it is in the three rows directly above current block 410, or if it is in the three columns directly to the left of current block 410, or if it is immediately adjacent to the left or right of the top row of current block 410. Spatially adjacent block 420 can be a rule size smaller than current block 410. For example, in... Figure 4 In the current block 410, it is a 32×32 block, and each adjacent block 420 is an 8×8 block.

[0055] Spatially adjacent blocks 420 can be examined to find one or more motion vectors (MVs) associated with the same reference frame index as the current block. For example, based on... Figure 4The block set described in the text is used to check the brightness blocks of spatially adjacent blocks, and the block set is marked according to the checking order. That is: (1) Check the top adjacent row from left to right. (2) Check the left adjacent column from top to bottom. (3) Check the top right adjacent block. (4) Check the top left adjacent block. (5) Check the first top non-adjacent row from left to right. (6) Check the first left non-adjacent column from top to bottom. (7) Check the second top non-adjacent row from left to right. (8) Check the second left non-adjacent column from top to bottom.

[0056] Candidates for “neighboring” spatial MV predictors derived from “neighboring” blocks (i.e., blocks in block sets 1-3) can be added to the MV predictor list before the candidates for temporal MV predictors in the temporal motion vector predictor (TMVP), as will be described further in this paper, and candidates for “non-neighboring” spatial MV predictors derived from “non-neighboring” blocks (also known as outer blocks, i.e., blocks in block sets 4-8) are added to the MV predictor list after the candidates for temporal MV predictors.

[0057] In one embodiment, each SMVP candidate has the same reference image as the current block. For example, suppose the current block 410 has a single reference image. If an MV candidate also has the same single reference image as the current block, then the MV candidate can be added to the MV predictor list. Similarly, if an MV candidate has multiple reference images, and one of those reference images is the same as the current block's reference image, then the MV candidate can be added to the MV predictor list. However, if the current block 410 has multiple reference images, then an MV candidate can be added to the MV predictor list only if the MV candidate has the same corresponding reference image for each of those reference images of the current block 410.

[0058] Figure 5 Illustrative operations of a set of Temporal Motion Vector Predictors (TMVPs) according to exemplary embodiments are described. TMVPs can be derived using juxtaposed blocks in reference frames. To generate a TMVP, firstly, one or more MVs of one or more reference frames can be stored together with reference indices associated with the respective reference frames. Subsequently, for each 8×8 block of the current frame, the MVs of reference frames whose trajectories cross the 8×8 block can be identified and stored in a temporal MV buffer along with the reference frame indices. For inter-frame prediction using a single reference frame, regardless of whether the reference frame is a "forward" or "backward" reference frame (i.e., later or earlier than the current frame in the frame sequence), the MV can be stored in 8×8 cells for performing temporal motion vector prediction for future frames. For composite inter-frame prediction, the MV of the "forward" reference frame can be stored in 8×8 cells for performing temporal motion vector prediction for future frames.

[0059] An exemplary embodiment of the process for generating a TMVP may follow the following operations. In this example, the reference motion vector 550 (also denoted as MVref) of the initial reference frame 510 points from the initial reference frame 510 to the subsequent reference frame 540, which is itself a reference frame of the initial reference frame 510. In doing so, the reference motion vector traverses the 8×8 block 570 (represented by gray dot shading) of the current frame 520. MVref 550 may be stored in a time MV buffer associated with this current block 570. During the motion projection process used to derive the time MV predictor 500, subsequent reference frames (e.g., frames 530 and 540) may be scanned in a predefined order. For example, using frame labels defined by the AV1 standard, the scan order may be: LAST_FRAME, BWDREF_FRAME, ALTREF_FRAME, ALTREF2_FRAME, and LAST2_FRAME. In one embodiment, MVs from higher-indexed reference frames (in scan order) do not replace previously identified MVs assigned by lower-indexed reference frames (in scan order).

[0060] Finally, given predefined block coordinates, the relevant MV stored in the time MV buffer can be identified and projected onto the current block 570 to derive a time MV predictor 560 (also labeled MV0) pointing from the current block 570 to the adjacent reference frame 530.

[0061] Figure 6 A set of illustrative predefined block positions 600 for deriving a time motion predictor for a 16×16 block, according to an exemplary embodiment, is depicted. The effective time motion predictor for up to seven blocks can be examined. Figure 6 In this context, these blocks are labeled B0-B6. (See reference...) Figure 4 The process involves examining candidates for the temporal MV predictor after those for adjacent spatial MV predictors but before those for non-adjacent spatial MV predictors, and placing them into a first MVP list. Then, to derive the MV predictor (MVP), all spatial and temporal MVP candidates can be aggregated, and each candidate can be assigned a weight determined during the scanning of spatially and temporally adjacent blocks. Based on the associated weights, the candidates can be sorted and ranked, and up to four candidates can be identified and placed into a second MVP list. This second MVP list, also known as the Dynamic Reference List (DRL), can be further used for dynamic MV prediction patterns.

[0062] If the DRL is not full, additional searches may be performed, generating additional MVP candidates to populate the DRL. These additional MVP candidates can include, for example, global MV, zero MV, composite MVs of combinations without scaling, etc. Then, adjacent SMVP candidates, TMVP candidates, and non-adjacent SMVP candidates in the DRL can be reordered again. Both AV1 and AVM support reordering, for example, based on the weight of each candidate. Candidate weights can be predefined based on the overlap area between the current block and candidate blocks.

[0063] Figure 7 Illustrative operations for generating new MV candidates via a single inter-frame prediction block, according to an exemplary embodiment, are described. When the reference frame of an adjacent block is different from the reference frame of the current block, but the MV is in the same direction, a time scaling algorithm can be used to scale the MV to that reference frame to form the MVP of the motion vector of the current block. Figure 7 In the example, the motion vector 740 from the neighboring block 750 of the current block 710 in the current frame 701 (in Figure 7 The middle part (also marked as mv1) points to the juxtaposed adjacent block 760 in reference frame 703. Motion vector 740 can be used to derive the motion vector 730 of the current block 710 using time scaling. Figure 7 The MVP (also marked as mv0) has its motion vector pointing to the current block 720 in another reference frame 702.

[0064] Figure 8 An illustrative operation for generating new MV candidates via a composite prediction block, according to an exemplary embodiment, is described. Figure 8 In the example, composite MVs 860 and 870 point from corresponding different neighboring blocks 820 and 830 of the current block 810 of the current frame 802 to reference frames 803 and 801. Composite MVs 860 and 870 (in Figure 8 Reference frames 803 and 801 (also marked as mv2 and mv3) can be the same as those in the current block 810. Composite inter-frame prediction can derive composite MVs 840 and 850 (in the current block 710) of the current block. Figure 8 The MVPs (also marked as mv0 and mv1) can be like... Figure 7 That's how it's determined.

[0065] Figure 9 Illustrative operations of updating a motion vector candidate library 920 according to exemplary embodiments are described. This library 920 was originally proposed in CWG-B023, the entire contents of which are incorporated herein.

[0066] The library update process can be based on superblock 910. That is, after encoding each superblock (e.g., superblock 910a), a set of first candidate MVs (e.g., the first 64 such candidate MVs) used by each encoded block within the superblock can be added to library 920. Pruning may also be involved during the update.

[0067] After completing a reference MV candidate scan of the superblock, if there are open slots in the candidate list, the codec can reference an additional MV candidate library 920 (in buffers with matching reference frame types). Starting from the end of the buffer and working backwards, if an MV in the library buffer does not exist in the candidate list, it can be appended to that candidate list. More specifically, each buffer can correspond to a unique reference frame type, corresponding to a single or a pair of reference frames, covering single and composite inter-frame modes respectively. All buffers can be the same size. When a new MV is added to a full buffer, existing MVs may be evicted to make room for the new MV.

[0068] The coded block can refer to the MV candidate library 920 to collect reference MV candidates, as well as the reference MV candidates generated through the AV1 reference MV list. After the superblock is encoded, the MV library can be updated using the MVs used by the coded blocks of the superblock.

[0069] AV1 can divide frames into tiles, where each tile contains multiple superblocks. Each tile can be processed in parallel on different processors. Regarding the candidate library, each tile can have an independent MV candidate library, which is used by all superblocks within the tile. When encoding each tile begins, the corresponding library is cleared. Subsequently, when encoding each superblock within that tile, MVs from the library can be used as MV reference candidates. After encoding each superblock, the library can be updated as described above.

[0070] Some embodiments of the library update and reference process used for library updates and references will be described later in this document.

[0071] Figure 10 This is a flowchart describing the process of constructing a motion vector prediction list for arbitrary video input according to an exemplary embodiment. Adjacent SMVP, TMVP, and non-adjacent SMVP candidates can be generated in S1010, S1020, and S1030 respectively, for example, by referring to previous examples. Figure 4 and Figure 5 The discussion process. Next, in S1040, for example, through previous references... Figure 6 The discussion process can involve sorting or reordering the candidates. In S1050, for example, this can be done by referring to previous references. Figure 7 and Figure 8The discussion process derives more MVP candidates. If needed, in S1060, for example, it can be done through previous references. Figure 6 The discussion process may involve identifying additional MVP candidates through further searches, or, in S1070, for example, through previous references. Figure 9 The discussion process involves retrieving additional MVP candidates from the reference library.

[0072] Figure 11 An illustrative operation of a composite inter-frame prediction mode according to an exemplary embodiment is described.

[0073] Composite inter-frame modes can create block predictions by combining hypotheses from multiple different reference frames. For example, in Figure 11 In the example, block 1111 of the current frame 1110 is composed of motion vectors 1130a and 1130b from adjacent reference frames 1120a and 1120b (in Figure 11 The prediction is also marked as mv0, mv1. Adjacent reference frames 1120a and 1120b can be direct neighbors (i.e., frames immediately before and after the current frame 1110 in the sequence), although this is not required. Motion information components for each block (e.g., motion vectors 1130a and 1130b) can be sent in the bitstream as overhead.

[0074] However, while motion vectors can often be predicted well using predictors from spatial and temporal neighbors or historical motion vectors, the bytes used for motion information can still be quite important for many content and applications.

[0075] Figure 12 An illustrative operation of a temporal interpolation prediction (TIP) mode according to an exemplary embodiment is described.

[0076] exist Figure 12 In the example, information from reference frames 1220a and 1220b is combined and projected onto the same point in time as the current frame 1210 using a simple interpolation process. Multiple TIP modes are supported. In one TIP mode, the interpolated frame, or "TIP frame" 1210', can be used as an additional reference frame. The coded blocks of the current frame 1210 can directly reference TIP frame 1210' and utilize information from two different references, requiring only the overhead of a single inter-frame prediction mode. In another TIP mode, TIP frame 1210' can be directly assigned as the output of the decoding process for the current frame 1210, skipping any other conventional coding steps. This mode offers significant coding and simplification advantages, especially for low bit-rate applications.

[0077] While existing techniques can interpolate frames between two reference frames, such as Frame Rate Upconversion (FRUC), achieving a good trade-off between complexity and compression quality can be a significant constraint when designing new encoding tools. The method disclosed above is simple and reuses motion information already available in the reference frames without requiring any additional motion search. Simulation results demonstrate that this simple method can achieve good quality with low complexity.

[0078] exist Figure 12 In the example, the TIP mode operation begins, generating a TIP frame 1210' corresponding to the current frame 1210. Then, TIP frame 1210' can be used as an additional reference frame for the current frame 1210, or directly allocated as the reconstruction output of the decoder for the current frame 1210. On the decoder side, blocks encoded in TIP mode can be generated on-the-fly, eliminating the need to create the entire TIP frame 1210' at the decoder, saving decoding time and processing. This also ensures compatibility with a single-pass decoding pipeline at the decoder, which is beneficial for hardware implementation.

[0079] Syntax elements can be used to indicate frame-level TIP modes. The following table shows examples of modes indicated by the tip_frame_mode parameter value.

[0080] tip_frame_mode meaning 0 Disable TIP mode in this framework 1 Using TIP frames as additional reference frames, the current frame is typically encoded. 2 Output the TIP frame directly, without encoding the current frame.

[0081] Table 1 – Examples of Tip Patterns

[0082] A simple interpolation method is disclosed for interpolating intermediate frames between two frames, which can fully reuse motion vectors from an available reference. The same motion vectors can also be used in the Temporal Motion Vector Predictor (TMVP) process with minor modifications. This process may involve three operations: 1. Creating a coarse motion vector field for the TIP frame by projecting the modified TMVP field. 2. Improving the coarse motion vector field by filling holes and using smoothing operations. 3. Generating the TIP frame using the improved motion vector field. On the decoder side, blocks encoded in the TIP mode can be generated on-the-fly without creating the entire TIP frame.

[0083] However, it should be noted that other suitable interpolation methods can be substituted in conjunction with other features discussed in this disclosure, and this is within the scope of this disclosure.

[0084] Figure 13 An illustrative operation of a decoder-side motion vector improvement based on bidirectional matching, according to an exemplary embodiment, is described. Universal Video Coding (VVC) can distribute previously decoded images to two reference image lists 1320a and 1320b. These previously decoded images can be used as a reference 1310 for predicting the current image. Figure 13In the example, reference images preceding the current image 1310 can be assigned to the "past" reference image list 1320a, while reference images following the current image 1320 can be assigned to the "future" reference image list 1320b, based on the display order. The corresponding reference image index (not shown) for each list indicates which image in each list is used to predict the current block 1311 of the current image 1310. For bidirectional prediction, the two prediction blocks 1321a and 1321b predicted using the corresponding MVs 1331a and 1331b of the past reference image list 1320a and future reference image list 1320b can be combined to obtain a single prediction signal.

[0085] If motion information is encoded using a merging pattern, the reference image index and MV of adjacent blocks can be directly applied to the current block 1311. However, this may not accurately predict the current block 1311.

[0086] The Decoder-Side Motion Vector Improvement (DMVR) algorithm can be used to improve the accuracy of merged mode coded blocks by involving only decoder-side information. When the DMVR algorithm is applied to blocks 1311, 1321a, and 1321b, the MVs 1331a and 1331b derived from the merged mode can be set as the “initial” MVs of the DMVR.

[0087] Then, DMVR can further improve the initial MVs 1331a and 1331b through block matching. In the two reference images, candidate blocks pointing to the initial MVs 1321a and 1321b can be searched to perform bidirectional matching. The best-matched blocks 1323a and 1323b can be used to generate the final prediction signal, and the new MVs 1333a and 1333b pointing to these new prediction blocks 1323a and 1323b can be set as "improved" MVs corresponding to the initial MVs 1331a and 1331b, respectively. Many block matching methods suitable for DMVR have been studied, such as template matching, bidirectional template matching-based methods, and bidirectional matching-based methods used in VVC.

[0088] In a bidirectional matching-based DMVR, the block pair 1321a and 1321b pointed to by the initial MV can be defined as the initial block pair. The distortion cost of the initial block pair 1321a and 1321b can be calculated as the initial cost. The blocks surrounding the initial block pair 1321a and 1321b can be used as DMVR candidate block pairs. Each block pair can include a predicted block from a reference image in the past reference image list 1320a and a predicted block from a reference image in the future reference image list 1320b.

[0089] The distortion cost of DMVR candidate block pairs can be measured and compared. Since the DMVR candidate block pair with the lowest distortion cost comprises the two most similar blocks between the reference images, it can be assumed that this block pair (i.e., blocks 1323a, 1323b) is the best predictor for the current block 1311. Therefore, block pair 1323a, 1323b can be used to generate the final bidirectional prediction signal. The corresponding MVs 1333a, 1333b can be represented as improved MVs. If all DMVR candidate block pairs have a higher distortion cost than the initial block pair 1321a, 1321b, then the initial blocks 1321a, 1321b can be used for bidirectional prediction, and the improved MVs 1333a, 1333b can be set equal to the initial MVs 1331a, 1331b.

[0090] To simplify distortion cost calculation, the sum of absolute differences (SAD) can be used as a distortion metric, and only luminance distortion can be considered during the DMVR search process. Note that SAD can be evaluated between even rows of candidate block pairs to further reduce computational complexity.

[0091] exist Figure 13 In the example, the dashed blocks (1321a, 1321b) in each reference image indicate the initial block pair. The gray blocks (1323a, 1323b) indicate the best-matching block pair, which can be the block pair with the lowest SAD cost compared to other DMVR candidate block pairs and the initial block pair 1321a, 1321b. The initial MVs 1331a, 1331b can be improved to generate improved MVs 1333a, 1333b, and the best-matching block pair 1323a, 1323b can be used to generate the final bidirectional prediction signal. Note that the initial MVs 1331a, 1331b do not need to point to the full sample location, as they can be derived from the merged mode, thus supporting up to 1 / 16 fraction of a sample MV accuracy.

[0092] Because the improved MV and the corresponding initial MV (in) Figure 13 The difference between ΔMV (1335a, 1335b) can be an integer or a fraction, so the improved MV can point to a fractional pixel position. In this case, intermediate search blocks and final prediction blocks can be generated through the DMVR interpolation process.

[0093] In some embodiments, a block-level bidirectional matching-based DMVR can be performed on top of the sports field generated by the TMVP. An example of this process will now be described with reference to the concepts previously described herein.

[0094] The process can begin by generating a motion field as part of the tip for each 8×8 block. The motion field is a representation of three-dimensional motion projected onto a two-dimensional space (e.g., an image) and is typically defined by one or more motion vectors, each describing the motion of a corresponding point. Here, the motion field may contain two motion vectors (MV0 and MV1) pointing to two reference images. The motion vectors (MV0 and MV1) can be used as the starting point for the DMVR process. More specifically, corresponding predictors in the reference images pointed to by the motion vectors can be generated. In this operation, filters (e.g., interpolation, bilinear, etc.) can be used to filter the input. Afterward, candidate predictors around the motion vectors can be generated. These predictors can be searched using a predefined search range N, which is an integer value corresponding to the number of luminance samples. The search precision is defined as K, which can be a fraction from 1 / 16, 1 / 8, 1 / 4, 1 / 2, 1, 2, 4, 8 to the number of luminance samples (maximum supported is MV precision). In the next operation, a bidirectional matching can be performed among all candidate predictors, and the position of the predictor containing the lowest distortion cost can be determined as the improved position of the 8×8 block. The distortion cost can be, but is not limited to, SAD, SATD, SSE, quadratic sampling SAD, mean-removed SAD, etc.

[0095] After obtaining the improved position (improved motion vector) for each 8×8 block, the TIP process can be performed. More specifically, the improved motion vector field of the DMVR can be used to generate TIP frames. The generated frames can be used as a reference for prediction or can be used directly for prediction.

[0096] On the decoder side, when blocks are encoded as TIPs or encoded via TIP mode, the TIP predictor and DMVR refinement can be performed on the fly without generating the entire frame. In some embodiments, sub-block-level bidirectional matching-based DMVR can be performed on top of the motion field generated by the TMVP. More specifically, for each set of 8×8 TIP blocks, additional segmentation can be performed. This additional segmentation can generate four 4×4 sub-blocks from each 8×8 block. Each sub-block can perform a bidirectional matching-based DMVR search to obtain the improved motion field of the TIP. Furthermore, sub-block-level bidirectional matching-based DMVR can be performed on top of the motion field generated by the TMVP and the optical flow refinement. More specifically, for each 8×8 TIP block, further segmentation can be performed. For example, each 8×8 TIP block can be divided into four 4×4 sub-blocks, where optical flow refinement is first applied to refine the motion vectors, and then a bidirectional matching-based DMVR search is further applied to refine the motion field of the TIP.

[0097] In some embodiments, sub-block-level bidirectional matching-based DMVR can be performed on top of the motion field generated by the TMVP and optical flow improvements. For example, for each 8×8 TIP block, an additional segmentation operation can generate four 4×4 sub-blocks, and a bidirectional matching-based DMVR improvement can be applied to improve the motion vector, followed by further optical flow improvements to improve the TIP's motion field. In some embodiments, multi-stage DMVR can be used to improve the TIP motion field generated by the TMVP. For example, a first block-level DMVR can be used to improve the generated initial motion field. The improved MV can be used as the starting point for a second stage. In the second stage, sub-block-level DMVR can be performed to further improve the motion field. Such additional stages are within the scope of this disclosure.

[0098] In other embodiments, the TIP motion field can use explicit signaled MV differences and / or corrections. Starting at any level (e.g., a set of coded blocks, a single coded block, or a sub-block level), one or more motion vector differences (MVDs) can be signaled into the bitstream. The decoder can parse the bitstream and use it as a correction for the TIP motion field. If a block is encoded in TIP mode, a TMVP-based approach can be used to generate the corresponding motion field for that block. The parsed MVDs can then be added to the motion field so that if the block is 8×8 or smaller, the block's MVs can be corrected using the parsed MVDs. If the block is larger than 8×8, each MV of each 8×8 sub-block can be added to the parsed MVDs.

[0099] In some embodiments, when applying a TIP using two reference images for motion compensation, MVD can be signaled to correct the motion field associated with the selected reference image. For example, MVD can be signaled for a future list of reference images, but not for a past list of reference images, and vice versa. Further signaling or implicit derivation can be used to determine which reference image requires further MVD signaling.

[0100] Figure 14 An illustrative use case of a merging pattern with Motion Vector Difference (MMVD) according to an exemplary embodiment is described. The merging pattern is typically used in conjunction with implicitly derived motion information to predict samples generated by the current coding unit (CU). The merging pattern with MMVD can use flags to signal MMVD for the CU. The MMVD flag can be sent after a skip flag is sent. In MMVD, after selecting a merging candidate, it can be further refined by signaled MVD information. This further information may include a merging candidate flag, an index specifying the motion amplitude, and an index indicating the motion direction. In MMVD mode, one of the top two candidates in the merging list can be selected as the MV basis. The merging candidate flag signals which candidate to use.

[0101] This operation can use a distance index that specifies motion amplitude information and indicates a predefined offset from the starting point. The offset can be added to the horizontal or vertical component of the starting MV.

[0102] Table 2 specifies example relationships between distance indices and predefined offsets.

[0103]

[0104] Table 2 – Relationship between Distance Index and Predefined Offset

[0105] The direction index can represent the direction of MVD relative to the starting point. The direction index can represent one of four directions, as shown in Table 3:

[0106] Directional IDX 00 01 10 11 x-axis + – N / A N / A y-axis N / A N / A + –

[0107] Table 3 – Sign of MV Offset Specified by Direction Index

[0108] The meaning of the MVD notation may vary depending on the information of the starting MV. When the starting MV is a one-way or two-way predictive MV, and both reference picture lists point to the same side of the current picture (i.e., the point of view (POC) of both references is greater than or less than the POC of the current picture), the notation in Table 3 can specify the sign of the MV offset added to the starting MV. When the starting MV is a two-way predictive MV, and both MVs point to different sides of the current picture (i.e., one reference has a POC greater than the POC of the current picture, and the other reference has a POC less than the POC of the current picture), and the POC difference in the first reference picture list is greater than the POC difference in the second reference picture list, the notation in Table 3 can specify the sign of the MV offset added to the first list MV component of the starting MV, while the sign of the second list MV can have the opposite value. Otherwise, if the POC difference in the second list is greater than the POC difference in the first list, the notation in Table 3 can specify the sign of the MV offset added to the second list MV component of the starting MV, while the sign of the first list MV can have the opposite value.

[0109] MVD can be scaled based on the difference in Proof of Conformity (POC) in each direction. If the POC differences in both lists are the same, scaling can be omitted. Otherwise, if the POC difference in one list is greater than the POC difference in the other list, the MVD of the list with the smaller POC difference can be scaled. If the initial MV is unidirectionally predicted, the MVD can be added to the available MVs.

[0110] In addition to unidirectional and bidirectional prediction MVD signaling, a symmetric MVD mode for bidirectional MVD signaling can also be applied. In the symmetric MVD mode, no signaling is performed, but motion information for MVD, including reference image indices for two reference image lists and a future reference image list, is derived.

[0111] In a specific implementation, the decoding process of the symmetric MVD mode can be as follows.

[0112] At the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 can be derived as follows: If mvd_l1_zero_flag is 1, then BiDirPredFlag is set to 0. Otherwise, if the most recent reference image in the past reference image list L0 and the most recent reference image in the future reference image list L1 form a forward and backward pair of reference images or a backward and forward pair of reference images, then BiDirPredFlag is set to 1, and both L0 and L1 reference images are short-term reference images. Otherwise, BiDirPredFlag is set to 0.

[0113] At the CU level, if the CU is bidirectional predictive coded and BiDirPredFlag equals 1, the symmetric mode flag can be explicitly signaled to indicate whether to use the symmetric mode. When the symmetric mode flag is true, mvp_l0_flag, mvp_l1_flag, and MVD0 can be explicitly signaled, while other signals can be omitted. The reference indices of L0 and L1 can be set to equal the reference picture pair, and MVD1 can be set to equal to (-MVD0).

[0114] In some embodiments, for each coded block in an inter-frame frame, if the current block's mode is not skip mode but inter-frame coding mode, another flag can be signaled to indicate whether a single reference mode or a composite reference mode is used for the current block. A prediction block can be generated from a single motion vector in single reference mode, and can be generated in composite reference mode by a weighted average of two prediction blocks derived from two motion vectors.

[0115] For a single reference pattern, the following specific patterns can be signaled according to the syntax of the example implementation:

[0116] NEARMV—A motion vector predictor (MVP) in a list indicated by a DRL (Dynamic Reference List) index.

[0117] NEWMV – Uses a motion vector predictor (MVP) from a list of DRL indexed signaling as a reference and applies an increment to the MVP.

[0118] GLOBALMV—Using motion vectors based on frame-level global motion parameters

[0119] For compound reference patterns, the following specific patterns can be signaled according to the syntax of the example implementation:

[0120] NEAR_NEARMV — A motion vector predictor (MVP) in a list that uses DRL indexing signaling.

[0121] NEAR_NEWMV — Uses a motion vector predictor (MVP) from the list of DRL indexed signaling as a reference and sends the incremental MV of the second MV.

[0122] NEW_NEARMV — Uses a motion vector predictor (MVP) from the list of DRL indexed signaling as a reference and sends the incremental MV of the first MV.

[0123] NEW_NEWMV — Uses a motion vector predictor (MVP) from the list of DRL indexed signaling as a reference and sends incremental MVs of two MVPs.

[0124] GLOBAL_GLOBALMV — MV used for each reference based on frame-level global motion parameters.

[0125] In some embodiments, this operation can allow 1 / 8 pixel motion vector precision (or accuracy), and in the example implementation, the following syntax can be used to signal the motion vector difference in L0 or L1:

[0126] mv_joint specifies which components of the motion vector difference are non-zero.

[0127] 0 indicates that there is no non-zero MVD along the horizontal or vertical direction.

[0128] 1 indicates that there is a non-zero MVD only in the horizontal direction.

[0129] 2 indicates that it has non-zero MVD only in the vertical direction.

[0130] 3 indicates that there is non-zero MVD in both the horizontal and vertical directions.

[0131] mv_sign specifies whether the motion vector difference is positive or negative.

[0132] `mv_class` specifies the class of motion vector difference. As shown in Table 4, a higher class indicates that the motion vector difference has a larger magnitude.

[0133]

[0134]

[0135] Table 4: Amplitude Levels of Motion Vector Difference

[0136] mv_bit specifies the integer part of the offset between the motion vector difference and the starting amplitude for each MV category.

[0137] mv_fr specifies the first two fractional bits of the motion vector difference.

[0138] mv_hp specifies the third fractional bit of the motion vector difference.

[0139] For NEW_NEARMV and NEAR_NEWMV modes, the precision of MVD may depend on the relevant category and the magnitude of the MVD. For example, fractional MVD is only allowed if the MVD magnitude is equal to or less than one pixel. Furthermore, when the value of the relevant MV category is equal to or greater than MV_CLASS_1, only one MVD value may be allowed, and the MVD value in each MV category is derived as 4, 8, 16, 32, and 64 for MV category 1 (MV_CLASS_1), 2 (MV_CLASS_2), 3 (MV_CLASS_3), 4 (MV_CLASS_4), or 5 (MV_CLASS_5).

[0140] Based on the above embodiments, Table 5 shows the allowed MVD values ​​for each MV category.

[0141]

[0142]

[0143] Table 5: Adaptive MVD in each MV amplitude category

[0144] Furthermore, if the current block is encoded in NEW_NEARMV or NEAR_NEWMV mode, one context can be used for signaling mv_joint or mv_class. Otherwise, another context can be used for signaling mv_joint or mv_class.

[0145] A new inter-frame coding mode named JOINT_NEWMV can be used to indicate whether the MVDs of the two reference lists are jointly signaled. If the inter-frame prediction mode is equal to the JOINT_NEWMV mode, then the MVDs of L0 and L1 signaling can be jointly signaled. More specifically, only one MVD (named joint_mvd) can be signaled and transmitted to the decoder, from which the incremental MVs of L0 and L1 can be derived.

[0146] The JOINT_NEWMV mode can be signaled together with NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. No additional context is required. When signaling the JOINT_NEWMV mode, and the POC distances between the two reference frames and the current frame are different, the MVD can be scaled based on the POC distance of L0 or L1. Let td0 be the POC distance between L0 and the current frame, and let td1 be the POC distance between L1 and the current frame. If td0 is equal to or greater than td1, then joint_mvd can be directly used for L0, and the MVD of L1 can be derived from joint_mvd based on equation (1):

[0147]

[0148] Otherwise, if td1 is equal to or greater than td0, then joint_mvd can be directly used for L1, and the mvd of L0 can be derived from joint_mvd based on equation (2):

[0149]

[0150] (Obviously, if td0 and td1 are equal, then according to either of the above equations, derived_mvd = joint_mvd; therefore, joint_mvd can be directly used as the derived mvd of L0 and L1, in which case scaling will not occur.)

[0151] Inter-frame coding mode (referred to here as AMVDMV mode) can be used for a single reference case. In AMVDMV mode, adaptive MVD (AMVD) resolution is applied to the signal MVD.

[0152] A flag (marked as amvd_flag here) can be added in JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode; this can be called joint AMVD coding. In joint AMVD coding, the MVDs of two reference frames can be jointed for signaling, and the precision of the MVD can be implicitly determined by the MVD amplitude. Otherwise, the MVDs of two (or more) reference frames can be jointed for signaling, and MVD coding can be applied.

[0153] The Adaptive Motion Vector Resolution (AMVR), originally proposed in CWG-C012 (the entire contents of which are incorporated herein by reference), supports seven MV precision values ​​(8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8). For each prediction block, the Adaptive Motion Vector (AVM) encoder can search for all supported precision values ​​and signal the optimal precision to the decoder.

[0154] To reduce encoder runtime, two precision sets can be supported. Each precision set can contain four predefined precisions. The precision set can be adaptively selected at the frame level based on the maximum precision value of the frame. As with standard AV1, the maximum precision can be signaled in the frame header. The following table summarizes the supported precision values ​​according to the maximum precision at the frame level.

[0155] Frame-level maximum precision Supported MV precision 1 / 8 1 / 8、1 / 2、1、4 1 / 4 1 / 4、1、4、8

[0156] Table 6: Supported MV Precision in the Two Sets

[0157] The AOMedia AVM library associated with AV1 provides frame-level flags to indicate whether the frame's MV includes subpixel precision. In some implementations, AMVR can only be enabled when the value of the `cur_frame_force_integer_mv` flag is 0. If the block's precision is below the maximum precision, the motion model and interpolation filters may remain unsignaled and inactive. If the block's precision is below the maximum precision, the applicable motion model can be inferred as a translation motion model, and the applicable interpolation filters can be inferred as "normal" filters. If the block's precision is 4 pixels or 8 pixels, the inter-frame mode can remain unsignaled and can be inferred as 0.

[0158] Figure 15 This diagram illustrates example components of a device or system 1500 on which embodiments of the systems and / or methods described herein may be implemented. The exemplary system 1500 can be one of a variety of systems, such as a personal computer, mobile device, computer cluster, server, embedded device, ASIC, microcontroller, or any other device capable of running code. A bus 1510 connects the exemplary system 1500 together, enabling all components to communicate with each other. The bus 1510 connects a processor 1520, a memory 1530, a storage component 1540, an input component 1550, an output component 1560, and an interface component.

[0159] Processor 1520 can be a single processor, a processor with multiple processors internally, a processor cluster (more than one), and / or distributed processing. The processor executes instructions stored in memory 1530 and storage component 1540. Processor 1520 operates as a computing device, performing operations to modify a shared cache of data exported by the Unreal Engine. Memory 1530 is fast memory, and retrieval of any memory device can be achieved through the use of cache memory, which can be closely associated with one or more CPUs. Storage component 1540 can be one of any long-term storage such as HDD, SSD, tape, or any other long-term storage format.

[0160] Input component 1550 can be any file type or signal from the user interface component, such as a camera, handheld controller, gamepad, keyboard, mouse, or input capture device, such as a motion capture device. Output component 1560 outputs processed information to communication interface 1570. The communication interface can be a speaker or other communication device, such as a screen, which can display information to a user or another observer (e.g., another computing system).

[0161] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Modifications and variations are possible based on the foregoing disclosure, or can be obtained through practical implementation.

[0162] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail integration. Furthermore, one or more of the aforementioned components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or multiple media) having computer-readable program instructions thereon for causing the processor to perform operations.

[0163] A computer-readable storage medium can be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical encoding devices (e.g., punched cards or raised structures in recesses storing instructions), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0164] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices or to external computers or external storage devices via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). This network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0165] Computer-readable program code / instructions used to perform the operations of this invention can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as C or similar languages). The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network (including local area network (LAN) or wide area network (WAN)) or can be connected to an external computer (e.g., via the Internet provided by an Internet service provider). In some embodiments, electronic circuitry (including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), and programmable logic arrays (PLAs)) can perform aspects or operations by utilizing the status information of the computer-readable program instructions to personalize the electronic circuitry.

[0166] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that the instructions, which execute via the processor of the computer or the processor of the other programmable data processing apparatus, perform the operations specified in the flowchart and / or block diagram. The computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium containing the instructions includes an article of manufacture comprising instructions for implementing aspects of the operations specified in one or more blocks of the flowchart and / or block diagram.

[0167] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented flow, such that the instructions executed on the computer, other programmable apparatus or other device perform the operations specified in one or more blocks of a flowchart and / or block diagram.

[0168] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical operation. The method, computer system, and computer-readable medium may include additional blocks, fewer blocks, different blocks, or blocks arranged differently from those shown in the figures. In some alternative embodiments, the operations shown in the blocks may not occur in the order shown in the figures. For example, in practice, depending on the functionality involved, two consecutively shown blocks may be executed simultaneously or substantially simultaneously, or sometimes these blocks may be executed in reverse order. It will also be noted that each block in the block diagram and / or flowchart illustrations, and combinations of blocks in the block diagram and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified operation or action or executes a combination of dedicated hardware and computer instructions.

[0169] Clearly, the systems and / or methods described herein can be implemented in various forms, including hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit these embodiments. Therefore, this document describes the operation and behavior of the systems and / or methods without referring to any specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

Claims

1. A decoding method for encoded video bitstreams, characterized in that, The method includes: Receive an encoded video bitstream including a current image, the current image including at least one block and a syntax element, the syntax element indicating that the at least one block should be predicted in a temporal interpolation prediction (TIP) mode; The motion field for the at least one block is generated using a time-based motion vector predictor (TMVP) method. The motion vector difference (MVD) parsed from the encoded video bitstream is added to the corresponding motion field to obtain the motion field generated for the at least one block. The motion field includes a first motion vector pointing to a first reference image and a second motion vector pointing to a second reference image, wherein the parsed MVD is used to correct the corresponding motion field. Based on the bidirectional matching process, a first improved motion vector and a second improved motion vector are generated by using the first motion vector and the second motion vector during the decoder-side motion vector enhancement (DMVR) process for the at least one block, without generating the entire frame; and The at least one block is decoded using the first improved motion vector and the second improved motion vector.

2. The method according to claim 1, characterized in that, The bidirectional matching process determines a first set of candidate motion vectors around the first motion vector and a second set of candidate motion vectors around the second motion vector. Wherein, the first improved motion vector is the candidate motion vector with the lowest distortion cost relative to the first motion vector among the first group of candidate motion vectors, and The second improved motion vector is the candidate motion vector with the lowest distortion cost relative to the second motion vector among the second group of candidate motion vectors.

3. The method according to claim 2, characterized in that, The method further includes: determining at least one candidate motion vector based on a search range N, where N is an integer value corresponding to the number of luminance samples; or determining at least one candidate motion vector based on a search precision K.

4. The method according to claim 3, characterized in that, The search precision K is a fractional value corresponding to the number of luminance samples; or the search precision K is an integer value corresponding to the number of luminance samples.

5. The method according to claim 1, characterized in that, The generation of the first improved motion vector and the second improved motion vector includes: Divide the at least one block into multiple sub-blocks, and The DMVR process is performed on each of the plurality of sub-blocks.

6. The method according to claim 5, characterized in that, The method further includes: For each of the plurality of sub-blocks, an optical flow improvement process is performed on the sub-block before the DMVR process is performed on the sub-block; or For each of the plurality of sub-blocks, after performing the DMVR procedure on the sub-block, an optical flow improvement procedure is performed on the sub-block.

7. The method according to claim 1, characterized in that, The generation of the first improved motion vector and the second improved motion vector includes: The DMVR procedure is performed on at least one block; After performing the DMVR procedure on the at least one block, the at least one block is divided into multiple sub-blocks; and The DMVR process is performed on each of the plurality of sub-blocks.

8. A video decoder, characterized in that, include: At least one communication module, the at least one communication module being configured to receive a bit stream; At least one non-volatile memory, the at least one non-volatile memory being configured to store computer program code; as well as At least one processor, connected to the at least one communication module and the at least one non-volatile memory, the at least one processor being configured to operate according to instructions of computer program code, the computer program code including: A receive code configured to cause at least one of the at least one processors to receive, via the at least one communication module, an encoded video bitstream including a current image, the current image including at least one block and a syntax element indicating that the at least one block should be predicted in a temporal interpolation prediction (TIP) mode; Generate code, which is configured to cause at least one of the at least one processors to use a time-based motion vector predictor (TMVP) method to generate a corresponding motion field for the at least one block, adding motion vector difference (MVD) parsed from the encoded video bitstream to the corresponding motion field to obtain a motion field generated for the at least one block, the motion field including a first motion vector pointing to a first reference image and a second motion vector pointing to a second reference image, wherein the parsed MVD is used to correct the corresponding motion field; Improved code, configured to cause at least one of the at least one processors to generate a first improved motion vector and a second improved motion vector based on a bidirectional matching process, by using the first motion vector and the second motion vector during decoder-side motion vector improvement (DMVR) for the at least one block without generating the entire frame; and Decoding code, the decoding code being configured to cause at least one of the at least one processors to decode the at least one block using the first improved motion vector and the second improved motion vector.

9. The video decoder according to claim 8, characterized in that, The improved code is also configured to cause at least one of the at least one processors to determine, through the bidirectional matching process, a first set of candidate motion vectors around the first motion vector and a second set of candidate motion vectors around the second motion vector. Wherein, the first improved motion vector is the candidate motion vector with the lowest distortion cost relative to the first motion vector among the first group of candidate motion vectors, and The second improved motion vector is the candidate motion vector with the lowest distortion cost relative to the second motion vector among the second group of candidate motion vectors.

10. The video decoder according to claim 9, characterized in that, The method further includes: determining at least one candidate motion vector based on a search range N, where N is an integer value corresponding to the number of luminance samples; or determining at least one candidate motion vector based on a search precision K, where the search precision K is one of a fractional value of the number of luminance samples and an integer value corresponding to the number of luminance samples.

11. The video decoder according to claim 8, characterized in that, The improved code is also configured to enable at least one of the at least one processors to: Divide the at least one block into multiple sub-blocks, and The DMVR process is performed on each of the plurality of sub-blocks.

12. The video decoder according to claim 11, characterized in that, The improved code is also configured to enable at least one of the at least one processors to: For each of the plurality of sub-blocks, an optical flow improvement process is performed on the sub-block before the DMVR process is performed on the sub-block; or The improved code is also configured to cause at least one of the at least one processor to perform an optical flow improvement process on each of the plurality of sub-blocks after performing the DMVR process on the sub-block.

13. The video decoder according to claim 8, characterized in that, The improved code is also configured to enable at least one of the at least one processors to: The DMVR procedure is performed on at least one block. After performing the DMVR procedure on the at least one block, the at least one block is divided into multiple sub-blocks; as well as The DMVR process is performed on each of the plurality of sub-blocks.

14. A non-transitory computer-readable storage medium, characterized in that, It stores instructions executable by at least one processor to perform a method for decoding an encoded video bitstream, the method comprising: Receive an encoded video bitstream including a current image, the current image including at least one block and a syntax element, the syntax element indicating that the at least one block should be predicted in a temporal interpolation prediction (TIP) mode; The motion field for the at least one block is generated using a time-based motion vector predictor (TMVP) method. The motion vector difference (MVD) parsed from the encoded video bitstream is added to the corresponding motion field to obtain the motion field generated for the at least one block. The motion field includes a first motion vector pointing to a first reference image and a second motion vector pointing to a second reference image, wherein the parsed MVD is used to correct the corresponding motion field. Based on the bidirectional matching process, a first improved motion vector and a second improved motion vector are generated by using the first motion vector and the second motion vector during the decoder-side motion vector enhancement (DMVR) process for the at least one block, without generating the entire frame; and The at least one block is decoded using the first improved motion vector and the second improved motion vector.

Images