Subblock candidates for automatic block vector relocation or chain motion vector prediction

Abstract

A method of concatenated vector usage for sub-block coding is provided. A video codec receives data to be encoded or decoded as a current block containing a plurality of current sub-blocks. The video codec identifies a plurality of co-located sub-blocks in a co-located picture corresponding to the plurality of current sub-blocks based on motion offsets provided by a concatenated vector, the concatenated vector being a sum of at least two recursively traced vectors, each traced vector being a motion vector or a block vector. The video codec derives predictors for the plurality of current sub-blocks based on motion information of the plurality of co-located sub-blocks. The concatenated vector is derived based on the motion information of the co-located sub-blocks to derive a predictor for a current sub-block. The video codec encodes or decodes the current block by using the derived predictors for the plurality of current sub-blocks.

Description

[0001] Cross-referencing

[0002] This disclosure is part of a non-provisional application and claims priority to U.S. Provisional Patent Applications Nos. 63 / 622,097 and 63 / 690,332, filed January 18, 2024 and September 4, 2024, respectively. The entire contents of the above applications are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to video encoding and decoding. More specifically, this disclosure relates to a method for encoding pixel blocks via chained motion vector prediction. Background Technology

[0004] Unless otherwise stated herein, the methods described in this section are not prior art to the following claims, nor are they admitted as prior art simply because they are included in this section.

[0005] High-Efficiency Video Coding (HEVC) is an international video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion-compensated discrete cosine transform (DCT-like) coding architecture. The basic unit used for compression is called a coding unit (CU), which is a 2N×2N square pixel block. Each CU can be recursively divided into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).

[0006] Versatile Video Coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11. The input video signal is predicted using a reconstructed signal derived from coded image regions. The prediction residual signal is processed through block transform. The transform coefficients, along with other auxiliary information, are quantized and entropy-coded in the bitstream. The reconstructed signal is generated from the predicted signal and the reconstructed residual signal obtained by inverse transforming the dequantized transform coefficients. The reconstructed signal is further processed by in-loop filtering to remove coding artifacts. The decoded image is stored in a frame buffer and used to predict subsequent images in the input video signal.

[0007] In VVC, the encoded image is divided into non-overlapping square block regions represented by relevant coding tree units (CTUs). The leaf nodes of the coding tree correspond to coding units (CUs). The encoded image can consist of multiple slices, each containing an integer number of CTUs. The individual CTUs within a slice are processed in raster scan order. Bidirectional prediction (B) slices predict the sample values ​​for each block using intra-frame prediction or inter-frame prediction using at most two motion vectors (MVs) and a reference index. Predictive (P) slices predict the sample values ​​for each block using intra-frame prediction or inter-frame prediction using at most one motion vector and a reference index. Intra-frame (I) slices are decoded solely using intra-frame prediction.

[0008] CTUs can be partitioned into one or more non-overlapping coding units (CUs) using quadtree (QT) and nested multi-type-tree (MTT) structures to accommodate different local motion and texture characteristics. CUs can be further partitioned into smaller CUs using one of the following five partitioning types: quadtree partitioning, vertical binary tree partitioning, horizontal binary tree partitioning, vertical center-side ternary tree partitioning, and horizontal center-side ternary tree partitioning.

[0009] Each CU contains one or more prediction units (PUs). The prediction unit, along with the associated CU syntax, serves as the basic unit indicating prediction sub-information. The specified prediction process is used to predict the values ​​of associated pixel samples within the PU. Each CU may contain one or more transform units (TUs) to represent prediction residual blocks. A transform unit (TU) consists of a transform block (TB) for the luma component and two corresponding chroma component transform blocks, each TB corresponding to a residual sample block from a color component. Integer transforms are applied to the transform blocks. The level values ​​of the quantization coefficients are entropy-encoded in the bitstream along with other auxiliary information. Coding tree blocks (CTBs), coding blocks (CBs), prediction blocks (PBs), and transform blocks (TBs) are used to specify two-dimensional sample arrays of a single color component associated with the CTU, CU, PU, ​​and TU, respectively. Therefore, a CTU contains one luma CTB, two chroma CTBs, and associated syntax elements. CUs, PUs, and TUs have similar correspondences.

[0010] For each inter-frame predicted CU, motion parameters include motion vectors, reference image indices, reference image list usage indices, and additional information used to generate inter-frame predicted samples. Motion parameters can be signaled explicitly or implicitly. When a CU is encoded in skip mode, the CU is associated with a PU and has no significant residual coefficients, no encoded motion vector increments, or reference image indices. Merge mode specifies that the motion parameters of the current CU can be obtained from neighboring CUs, including spatial and temporal candidates, as well as additional scheduling introduced in VVC. Merge mode can be applied to any inter-frame predicted CU. An alternative to merge mode is explicit transmission of motion parameters, where the motion vectors of each CU, the corresponding reference image indices for each reference image list, the reference image list usage flags, and other necessary information are explicitly signaled.

[0011] Intra-block copy (IBC) or current picture referencing (CPR) refers to encoding pixel blocks by referencing pixel locations within the same current picture as the current block using block vector references.

[0012] In Advanced Motion Vector Prediction (AMVP) mode, motion vector predictor (MVP) candidates are determined based on template matching (TM) error. The candidate with the smallest difference between the current block template and the reference block template is selected, and then TM is performed only on that specific MVP candidate to refine the motion vectors. The TM process can refine the MVP candidate using an iterative search based on the Adaptive Motion Vector Resolution (AMVR) mode search pattern. Summary of the Invention

[0013] The following summary is for illustrative purposes only and is not intended to be limiting. That is, the following summary aims to introduce the concepts, key points, benefits, and advantages of the novel and non-obvious techniques described herein. Some, and not all, embodiments will be further described in detail below. Therefore, the following summary is not intended to identify the essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter.

[0014] Some embodiments of this disclosure provide a concatenated vector method for sub-block coding. A video codec receives data to be encoded or decoded as the current pixel block of a current video image. This current block includes multiple current sub-blocks. The video codec identifies multiple co-located sub-blocks in the co-located image corresponding to the multiple current sub-blocks. Based on motion information of the multiple co-located sub-blocks, the video codec derives predictors for the multiple current sub-blocks. The video codec uses the derived predictors of the multiple current sub-blocks to encode or decode the current block.

[0015] In some embodiments, when the width or height of the co-location image differs from the current image, the video codec applies positional scaling to the co-location image. In some embodiments, motion offsets are provided by a cascaded vector (or chained motion vector, chain MV), which is the sum of a recursively tracing motion vector and a block vector. The cascaded vector has a base vector, which is the motion vector or block vector of the neighboring block of the current block. In some embodiments, the motion offset candidate list includes: (i) the block vector or motion vector of the neighboring block, which is the base vector of the cascaded vector, and (ii) the cascaded vector.

[0016] In some embodiments, the motion information of the plurality of co-located sub-blocks is scaled based on the time distance between the current image, the co-located image, and the reference image of the motion information of the plurality of co-located sub-blocks, and the prediction of the plurality of current sub-blocks is derived.

[0017] In some embodiments, a concatenated vector (or chained motion vector) is derived based on motion information of co-located sub-blocks to derive a predictor for the current sub-block, the concatenated vector being the sum of recursively tracing motion vectors or block vectors. In some embodiments, the motion information of the co-located sub-block is time-scaled to the current image to be used as a basis vector for deriving the concatenated vector, and a reference block is identified from the current sub-block, the scaling being applied according to a scaling factor based on: (i) a first time difference between the co-located image and the reference image identified by the motion information of the co-located sub-block; and (ii) a second time difference between the current image and the reference image identified by the concatenated vector.

[0018] In some embodiments, motion information of the co-located sub-block is used as a basis vector for deriving the concatenated vector. The predictor of the current sub-block may be derived based on a reference block identified from the co-located sub-block by the concatenated vector. In some embodiments, the predictor of the current sub-block may be derived based on a scaled motion vector identified from the current sub-block by time scaling the concatenated vector to the current image, the scaling being applied according to a scaling factor calculated based on: (i) a first time difference between the co-located image and the reference image identified by the concatenated vector; and (ii) a second time difference between the current image and the reference image identified by the scaled motion vector. Attached Figure Description

[0019] The accompanying drawings are included in this disclosure to further understand it and form part of this disclosure. The drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure. It will be understood that the drawings are not necessarily drawn to scale, as some components may be shown out of proportion to actual implementations for clarity of the concepts of the disclosure.

[0020] Figure 1 The control point motion vector (CPMV) of the current block, encoded by the affine motion field, is shown.

[0021] Figure 2 The affine motion field for each sub-block is shown.

[0022] Figure 3 The candidate neighbor block used for inheriting affine candidates is shown.

[0023] Figure 4 This conceptually demonstrates the inheritance of control point motion vectors.

[0024] Figure 5A -B shows the spatial neighbors used to derive affine merging / AMVP candidates.

[0025] Figure 6 A candidate for affine merging / AMVP is shown, which transforms motion information from non-neighboring neighbors to a first-type construction.

[0026] Figure 7 The adjacent 4 x 4 sub-blocks used for RMVF parameter derivation are shown.

[0027] Figure 8 The diagram illustrates how the motion field of a sub-CU is derived by applying motion offsets from spatial neighbors and scaling motion information from the corresponding co-located sub-CU.

[0028] Figure 9 The derivation of the automatic relocation block vector is shown conceptually.

[0029] Figure 10 The concept illustrates the location to be checked for deriving the automatic relocation block vector.

[0030] Figure 11 An example derivation of a Chain Motion Vector Prediction (CMVP) candidate is shown.

[0031] Figure 12 The possible sources of the CMVP's basis vectors are shown.

[0032] Figure 13 This concept demonstrates how to derive sub-block motion by scaling motion from an image at the same location.

[0033] Figure 14A This concept demonstrates how to predict the current sub-block by referencing the sub-block at the same location in the same image.

[0034] Figure 14B This concept demonstrates how to use the reference block of the reference block of the same positional sub-block as the prediction of the current sub-block.

[0035] Figure 15 This concept illustrates scaling the chain motion vectors of images at the same location to the sub-block motion of the current block.

[0036] Figure 16 This shows the positional scaling of the image at the same location.

[0037] Figure 17 An example video codec that enables sub-block motion is shown.

[0038] Figure 18 The video codec section that implements sub-block processing and concatenated vectors is shown.

[0039] Figure 19 This conceptually illustrates the process of encoding sub-blocks using cascaded vectors.

[0040] Figure 20 An example video decoder that enables sub-block motion is shown.

[0041] Figure 21 The video decoder section that implements sub-block processing and concatenated vectors is shown.

[0042] Figure 22 This conceptually illustrates the process of decoding sub-blocks using cascaded vectors.

[0043] Figure 23 An electronic system that implements certain embodiments of this disclosure is conceptually illustrated. Detailed Implementation

[0044] In the following detailed description, numerous specific details are given by way of example to provide a thorough understanding of the related techniques. Any variations, derivatives, and / or extensions based on the techniques described herein are within the scope of this disclosure. In some cases, to avoid unnecessarily obscuring various aspects of the techniques disclosed, well-known methods, processes, components, and / or circuits relating to one or more exemplary embodiments may be described at a relatively high level without detail.

[0045] I. Affine Prediction

[0046] A. Affine Stadium

[0047] Objects in a video may exhibit different types of motion, including translation, scaling, rotation, perspective motion, and other irregular motions. In some embodiments, block-based affine transformation motion compensation prediction is employed to accommodate these different types of motion. Block-based affine transformation motion compensation prediction can be used. Specifically, the affine motion field of the current block at position (x, y) mv x , mv y Using a linear model:

[0048] The coefficients {a, b, c, d, e, f} are the parameters of this linear model. In some embodiments, the affine motion field at position (x, y) can be described by motion information from two control points (CPs) (e.g., the top right and top left corners of the block) (4-parameter model) or motion information from three control points (e.g., the top right, top left, and bottom left corners of the block) (6-parameter model).

[0049] For a 4-parameter affine motion model, eq.0 (the motion vector at sampling position (x, y) in the block) can be written as: (1) For a 6-parameter affine motion model, eq.0 can be written as: (2) Where (mv 0x , mv 0y ) is the motion vector of the top-left control point (top-left CPMV, or mv0), (mv 1x ,mv 1y ) is the motion vector of the upper right control point (upper right CPMV, or mv1), (mv 2x , mv 2y ) is the motion vector of the lower left control point (lower left CPMV, or mv2).

[0050] In some embodiments, a 2-parameter model can be used to refine the inter-frame translation prediction candidates (e.g., conventional candidate merging):

[0051] Figure 1 The control point motion vector (CPMV) of the current block is shown, encoded by an affine motion field. The current block has CPMVs at its top left (mv0), top right (mv1), and bottom left (mv2). The affine motion field mv' at position (x, y) in the current block 100 can be derived using affine motion models such as eq.1 (4-parameter affine model) or eq.2 (6-parameter affine model).

[0052] To simplify motion compensation prediction, block-based affine transformation prediction was applied. Figure 2 The affine motion field for each sub-block is shown. This figure illustrates the motion vector field of a block with 16 4x4 sub-blocks. To derive the motion vector for each 4×4 luma sub-block, the motion vector at the center sampling point of each sub-block is calculated according to eq.1 or eq.2 and rounded to 1 / 16 precision. A motion-compensated interpolation filter can be applied to generate a prediction for each sub-block using the derived motion vectors. The sub-block size for the chroma components is also set to 4×4. The MV of the 4×4 chroma sub-block is calculated as the average of the MVs of the upper-left and lower-right luma sub-blocks within the 8x8 luma region at the same location.

[0053] B. Affine Merging Pattern

[0054] In affine merging mode, the motion vector (CPMV) at the control point of the current coding unit (CU) is generated based on the motion information of spatially neighboring CUs. There can be up to five CPMVP candidates, and the candidate used for the current CU is indicated by an index. The following three types of CPMV candidates are used to form the affine merging candidate list: (1) inherited affine merging candidates extrapolated from the CPMVs of neighboring CUs; ​​(2) constructed affine merging candidate CPMVPs derived using the translation MVs of neighboring CUs; ​​and (3) zero MV.

[0055] The affine candidate is inherited by directly obtaining the CPMV from the neighboring block and inheriting the affine model from the neighboring block (one from the left neighboring CU and one from the top neighboring CU). Figure 3 The candidate neighbor blocks used for inheriting affine candidates are shown. For left-hand prediction, the scan order of the candidate blocks is A1. A0, for the above prediction, the scanning order of the candidate blocks is B1. B0 B2. When a neighboring affine CU is identified, its control point motion vector is used to derive CPMVP candidates in the affine merging list of the current CU.

[0056] Figure 4 This conceptually illustrates the inheritance of control point motion vectors. As shown in the figure, for the current block 410, if the lower-left neighboring block A is encoded in affine mode, the motion vectors mv2, mv3, and mv4 of the upper-left, upper-right, and lower-left corners of CU 420 containing block A can be inherited by the current block 410. When block A is encoded in a 4-parameter affine model, the two CPMVs of the current CU 410 can be calculated based on mv2 and mv3. If block A is encoded in a 6-parameter affine model, the three CPMVs of the current CU 410 can be calculated based on mv2, mv3, and mv4.

[0057] The constructed affine candidate refers to the process of building candidates by combining the translational motion information of each control point's neighbors. The motion information of the control points is derived from... Figure 3 The spatial and temporal neighbors shown are derived using CPMV. k (k=1, 2, 3, 4) represents the k-th control point. For CPMV1, check B2. B3 Block A2, and use the MV of the first available block. For CPMV2, check B1. For block B0, check A1 for CPMV3. Block A0. If TMVP is available, it is used as CPMV4. After obtaining the MVs of the four control points, affine merge candidates are constructed based on this motion information. The following combinations of control point MVs are used in sequence for construction: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}. Combinations of 3 CPMVs construct 6-parameter affine merge candidates, and combinations of 2 CPMVs construct 4-parameter affine merge candidates. To avoid motion scaling, if the reference indices of the control points are different, the relevant control point MV combinations are discarded. After checking the inherited affine merge candidates and the constructed affine merge candidates, if the list is still not full, a zero MV is inserted at the end of the list.

[0058] C. Affine AMVP Prediction

[0059] The affine AMVP mode can be applied to CUs with a width and height greater than or equal to 16. An affine flag is sent in the CU-level bitstream to indicate whether the affine AMVP mode is used, followed by another flag to indicate whether it is a 4-parameter affine or a 6-parameter affine. In this mode, the difference between the current CU's CPMV and its predicted CPMVP is sent in the bitstream. The affine AVMP candidate list is of size 2 and is generated by sequentially using the following four types of CPMV candidates: (1) inherited affine AMVP candidates extrapolated from the CPMV of neighboring CUs, (2) constructed affine AMVP candidates CPMVP derived using the translation MV of neighboring CUs, (3) translation MV of neighboring CUs, and (4) zero MV.

[0060] The checking order for inherited affine AMVP candidates is the same as that for inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference image as the current block are considered. No pruning is performed when inserting inherited affine motion predictors into the candidate list.

[0061] The constructed AMVP candidates are from the above. Figure 3 The spatial neighbor derivation is shown. The same checking order as the affine merge candidate construction is used. Additionally, the reference image index of neighboring blocks is checked. The first block in the checking order is inter-frame coded and uses the same reference image as the current CU. When the current CU is coded in 4-parameter affine mode and both mv0 and mv1 are available, they are added as a candidate to the affine AMVP list. When the current CU is coded in 6-parameter affine mode and all three CPMVs are available, they are added as a candidate to the affine AMVP list. Otherwise, the constructed AMVP candidate is set to unavailable.

[0062] If the affine AMVP candidate list still has fewer than two candidates after inserting valid inherited affine AMVP candidates and constructed AMVP candidates, then mv0, mv1, and mv2 will be added in sequence as translation MVs to predict all control point MVs of the current CU (if available). Finally, if the list is still not full, the affine AMVP candidate list is filled with zero MVs.

[0063] D. Affine model inheritance and non-adjacent affine modes based on historical parameters

[0064] History-parameter-based affine model inheritance (HAMI) allows affine models to inherit from previously affine-coded blocks, which may not be neighbors of the current block. Similar to the enhanced regular merging mode, it introduces a non-adjacent affine mode (NA-AFF). A first history-parameter table (HPT) is established. Each entry in the first HPT stores a set of affine parameters: a, b, c, and d, each represented by a 16-bit signed integer. Entry in the HPT is categorized by reference lists and reference indices. Each reference list supports five reference indices in the HPT. The HPT category (denoted as HPTCat) is calculated as follows: HPTCat (RefList, RefIdx) = 5×RefList + min (RefIdx, 4) RefList and RefIdx represent the list of reference images (0 or 1) and the reference index, respectively. A maximum of seven entries can be stored for each category, therefore a total of 70 entries can be stored in the HPT. At the beginning of each CTU line, the number of entries for each category is initialized to zero. Decoding involves using the reference list RefList... cur and RefIdx cur After affine encoding of the CU, the affine parameters are used to update the category HPTCat(RefList) in a manner similar to HMVP table updates. cur ,RefIdx cur ) entries.

[0065] History-affine-parameter-based candidate (HAPC) is derived from... Figure 3 The MV of the current block at position (x, y) is derived from one of the seven neighboring 4×4 blocks A0, A1, A2, B0, B1, B2, or B3, and a set of affine parameters stored in the corresponding entry in the first HPT. The MV of the neighboring 4×4 blocks is used as the reference MV. The MV of the current block at position (x, y) is calculated as follows: , in( mv h base , mv v base ) represents the MV of the adjacent 4×4 blocks, ( x base , ybase (x, y) represents the center position of the nearest 4×4 block. (x, y) can be the top left, top right, and bottom left corners of the current block to obtain the corner MV (CPMV) of the current block, or it can be the center of the current block to obtain the regular MV of the current block.

[0066] The second history-parameter table (HPT) also includes basis motion vector (MV) information. The second HPT contains nine entries, each including a basis motion vector, a reference index, four affine parameters for each reference list, and a base position. An additional merged HAPC can be generated based on the basis motion vector information stored in the second HPT and the corresponding affine model.

[0067] Furthermore, paired affine merge candidates are generated from two affine merge candidates, which can be either historically derived or non-historically derived. Paired affine merge candidates are generated by averaging the CPMV of existing affine merge candidates in the list. In response to the introduction of the new HAPC, the size of the sub-block-based merge candidate list has increased from five to fifteen, all of which participate in the ARMC process.

[0068] The method for acquiring non-adjacent spatial neighbors in non-adjacent affine mode (NA-AFF) is the same as that for existing non-adjacent regular merge candidates. The distance between non-adjacent spatial neighbors and the current coding block in NA-AFF is also defined based on the width and height of the current coding unit (CU).

[0069] Motion information from non-adjacent spatial neighbors is used to generate additional inheritance and construct affine merge / AMVP candidates. Specifically, for inheritance candidates, the derivation process for inheriting affine merge / AMVP candidates in VVC remains unchanged; the only difference is that the checkpoint motion vector is inherited from non-adjacent spatial neighbors. Non-adjacent spatial neighbors are checked based on their distance from the current block, i.e., from nearest to farthest. At a specific distance, only the first available neighbor (encoded in affine pattern) from each side (e.g., left and top) of the current block is included for the derivation of inheritance candidates. Figure 5A -B shows the spatial neighbors used to derive affine merge / AMVP candidates. Figure 5A The spatial neighbors used to derive inheritance candidates are shown. Figure 5B The spatial neighbors used to derive the first type of construction candidate are shown.

[0070] like Figure 5AAs shown by the dashed arrow, the inspection order for the left and top neighbors is bottom-to-top and right-to-left, respectively. For the first type of construction candidate, such as... Figure 5B As shown, first, the positions of a non-adjacent spatial neighbor to the left and above are independently determined. Then, the position of the top-left neighbor can be determined accordingly, and this neighbor, together with the non-adjacent neighbors to the left and above, can form a rectangular virtual block.

[0071] Figure 6 The transformation of motion information from non-adjacent neighbors to first-class construct affine merge / AMVP candidates is illustrated. As shown, the motion information of three non-adjacent neighbors is used to form the checkpoint motion vectors of the top-left (A), top-right (B), and bottom-left (C) corners of the virtual block, which are ultimately projected onto the current CU to generate the corresponding construct candidates. NA-AFF candidates are inserted into the existing affine merge candidate list and affine AMVP candidate list in the following order: Affine Merge Mode: (1) SbTMVP candidate (if available) (2) Inherited from neighboring neighbors (3) Inherited from non-adjacent neighbors (4) Construct from neighboring neighbors (5) Construct affine candidates from the first type of non-adjacent neighbors. (6) Zero motion vector Affine AMVP mode: (1) Inherited from neighboring neighbors (2) Constructing from neighboring neighbors (3) Translational motion vectors from neighboring neighbors (4) Translational motion vectors from time-domain neighbors (5) Inherited from a non-adjacent neighbor (6) Construct affine candidates from the first class of non-adjacent neighbors (7) Zero motion vector The size of the affine merge candidate list increases from 5 to 15 due to the inclusion of additional candidates generated by NA-AFF. The subgroup size of ARMC in affine merge mode increases from 3 to 15. In NA-AFF: (1) the regions from non-adjacent neighbors are restricted to the current CTU (i.e., no additional row buffer storage is required); (2) the storage granularity of affine motion information (including CPMV and reference index) is reduced from 8x8 to 16x16 (i.e., only the affine motion of the top-left 8x8 block is stored); in addition, the stored CPMV is projected onto each 16x16 block before storage, so position and size information is not required; (3) only the top-left and top-right CPMVs are stored (i.e., NA-AFF always uses a 4-parameter affine model).

[0072] E. Affine Candidate Derivation Based on Regression

[0073] The regression-based motion vector field (RMVF) derivation method provides a novel sub-block-based merging candidate. Figure 7 The diagram shows adjacent 4x4 sub-blocks used for RMVF parameter derivation. W and H represent the width and height of the current CU, respectively. As shown, the motion vectors and center positions of the current CU's adjacent sub-blocks are used as inputs to the linear regression process to derive a set of linear model parameters.

[0074] The sub-block motion field from the previously encoded affine CU and the motion vectors of the current CU's neighboring sub-blocks are used as inputs to the regression process. The predicted CPMV of the current block is derived as output. Affine merging candidates based on the regression are derived and added to the affine merging list. The sub-block motion field from the previously encoded affine CU and the motion information of the current CU's neighboring sub-blocks are used as inputs to the regression process to derive affine candidates.

[0075] Previously encoded affine CUs can be identified by scanning non-adjacent locations and the affine HMVP table. The adjacent sub-block information of the current CU is derived from the adjacent 4x4 sub-blocks (…). Figure 7 (The gray area represents the data). For each sub-block, given a list of references, the corresponding motion vector and center coordinates of that sub-block can be used.

[0076] For each affine CU, a maximum of two affine candidates can be derived, one containing information about neighboring subblocks and the other not. All candidates generated by linear regression are pruned and collected into a candidate subgroup, and when ARMC is enabled, the ARMC process based on TM cost is applied. Then, when N affine CUs are found, at most N candidates generated by linear regression are added to the affine merge list. ARMC has 30 affine candidates and an output list size of 15.

[0077] II. Subblock-based temporal motion vector prediction (SbTMVP)

[0078] VVC supports the Sub-Block-Based Temporal Motion Vector Prediction (SbTMVP) method. Similar to temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the same image to improve the motion vector prediction and merging pattern of CUs in the current image.

[0079] Similar to the co-location image used in Temporal Motion Vector Prediction (TMVP), Sub-block Temporal Motion Vector Prediction (SbTMVP) also uses the co-location image. SbTMVP differs from TMVP in two main ways: (1) TMVP predicts motion at the coding unit (CU) level, while SbTMVP predicts motion at the sub-CU level. (2) TMVP obtains temporal motion vectors from co-location blocks in the co-location image (co-location blocks are relative to the lower right corner or center of the current CU), while SbTMVP applies a motion offset before obtaining temporal motion information from the co-location image. This motion offset is obtained from the motion vector of one of the spatially neighboring blocks of the current CU (e.g., such as...). Figure 3 The spatial neighboring block A1 is shown.

[0080] SbTMVP predicts motion vectors for each sub-CU within the current CU in two steps. In the first step, spatially neighboring blocks (e.g., A1) are examined. If A1 has a motion vector with a co-location image as its reference image, that motion vector is selected as the motion offset to apply. If no such motion is identified, the motion offset is set to (0, 0). In the second step, the motion offset identified in step 1 (i.e., added to the coordinates of the current block) is applied to obtain sub-CU-level motion information (motion vector and reference index) from the co-location image.

[0081] Figure 8This illustrates how the motion field of a sub-CU is derived by applying motion offsets from spatially neighboring blocks and scaling motion information from corresponding sub-CUs at the same location. Figure 8 In the example, the motion offset is set to the motion of the spatially neighboring block A1. Then, for each sub-CU of the current block (i.e., the current sub-CU), the motion information of the current sub-CU is derived using the motion information of the corresponding block in the co-location image (i.e., the co-location sub-CU; the smallest motion grid covering the center sampling point). After identifying the motion information of the co-location sub-CU, it is converted into the motion vector and reference index of the current sub-CU, similar to the TMVP process in High Efficiency Video Coding (HEVC), where temporal motion scaling is applied to align the reference image of each temporal motion vector with the reference image of the current CU.

[0082] In Universal Video Coding (VVC), a sub-block-based merge list containing SbTMVP candidates and affine merge candidates is used for signaling in the sub-block-based merge mode. SbTMVP candidates (such as A1) can be referred to as motion offset candidates. The SbTMVP mode is enabled or disabled via the Sequence Parameter Set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is the first entry in the sub-block-based merge candidate list, followed by the affine merge candidate. The size of the sub-block-based merge list is signaled in the SPS (e.g., the maximum allowed size of the sub-block-based merge list in VVC is 5). The sub-CU size used in SbTMVP is fixed at 8x8. The SbTMVP mode is only applicable to CUs with both width and height ≥ 8.

[0083] The encoding logic for the additional SbTMVP merge candidate is the same as that for other merge candidates, that is, for each CU in the P-slice or B-slice, an additional rate-distortion (RD) check is performed to determine whether to use the SbTMVP candidate.

[0084] III. Cascaded Vector Prediction

[0085] A. Auto-relocated block vector prediction (AR-BVP)

[0086] Automatic relocation block vector prediction (AR-BVP) is part of the construction of the candidate list for intra block copy (IBC) merging / adaptive motion vector prediction (AMVP). Figure 9 The derivation of the automatic relocation block vector is conceptually illustrated. As shown in the figure, for the current image 900, the guide block vector BV associated with the current block B0 is... 0,1Point to reference block B1. If B1 has a block vector (BV) pointing to reference block B2, it is denoted as BV. 1,2 Then BV 0,2 By BV 0,2 = BV 0,1 + BV 1,2 Given, defined as being by BV 0,1 Guided AR-BVP. Similarly, BV 0,n+1 Through BV 0,n+1 = BV 0,n + BV n,n+1 = BV 0,1 + BV 1,2 + …+ BV n-1,n + BV n,n+1 Derivation.

[0087] In some embodiments, the length of the AR-BVP tracking path is 1 (i.e., n=1). In some embodiments, the length of the AR-BVP tracking path is 2 (i.e., n=2). In some embodiments, the length of the AR-BVP tracking path is not constrained.

[0088] Figure 10 The diagram conceptually illustrates the locations to be checked when deriving the automatic relocation block vector. As shown in the figure, in the BV... 0,n, Guided derivation of BV n,n+1 At that time, B can be checked. n The five positions, including top left (“LT”), top right (“RT”), center (“Ctr”), bottom left (“LB”), and bottom right (“RB”), are used to locate BV. n,n+1 In some embodiments, the initial boot block vector BV 0,1 The BVP is set to one that already exists in the IBC merge / AMVP candidate list. In some embodiments, AR-BVP candidates may be inserted after historical BVP candidates. The size of the IBC merge / AMVP candidate list may remain unchanged.

[0089] B. Chained motion vector prediction (CMVP)

[0090] In some embodiments, chained motion vector prediction (CMVP) is included in the construction of the inter-frame merging candidate list. Figure 11A derivation example of Chained Motion Vector Prediction (CMVP) candidates is shown. As illustrated, CMVP candidates can be derived based on motion vectors (MVs) pre-derived for the inter-frame merging candidate list, as the sum of the MVs and BVs of the recursively tracked vectors. The figure shows a CMVP candidate as motion vector MVL0. k / m It is located in the reference image aRefPicL0 k / m A block within [the image]. The motion vector and reference image are determined as follows: MVL0 k / m = MVL0 k(0) + BV k(0) + MVL0 k(1) + MVL0 k(2) + … + MVL0 k(m) , RefPicL0 k / m = RefPicL0 k(m) More generally, for CMVP candidates, the following formula can be used in the reference image RefPic k / m Derive a set of motion vectors MV k / m : MV k / m =MV k(0) + BV k(0) +MV k(1) +MV k(2) + … + MV k(m) , RefPic k / m =RefPic k(m) , Where k and m represent the number of merged indexes and the tracking depth of the CMVP, respectively.

[0091] Figure 12 The possible sources of the basis vectors for a CMVP are shown. The diagram illustrates the operation of referencing sources and destinations in a CMVP to trace the motion vector (MV). As shown, when deriving the MV... k / m At this time, the MV is found by checking whether the MV or block vector (BV) exists in the MV / BV storage corresponding to all five positions of the current block (i.e., the center (Ctr), top left (TL), top right (TR), bottom left (BL), and bottom right (BR) of the current block). k(0) (Also known as the base vector of CMVP). In this example, an MV is found in the MV / BV storage corresponding to the current block center location, and this MV is used as MVL0. k(0) Or the basis vector of CMVP.

[0092] When the pre-derived merge candidate for the CMVP candidate has two MVs, derive one MV for each list (i.e., L0 and L1) and for each tracking depth. k / m At most two MVs can be derived for each list and each tracking depth, and the MV set is inserted sequentially into the inter-frame merge candidate list. Trackable reference images are limited to the reference image list. For regular merge and TM merge, CMVP candidates can be inserted after HMVP candidates. When deriving CMVP candidates, hpelIfIdx, bcwIdx, licFlag, and mhpFlag may not be inherited. When TMVP is disabled, CMVP candidates may not be derived.

[0093] IV. Affine Prediction Based on Chain Motion Vectors

[0094] A. Affine inheritance candidate

[0095] In some embodiments, affine inheritance candidates can be derived by recursively tracing MV and BV based on an initial MV or BV (as a basis vector). Specifically, if a spatial candidate or non-adjacent candidate is not encoded affinely, the MV or BV of that spatial candidate is set as the initial MV or BV of the Chained Motion Vector Prediction (CMVP) to derive the chained motion vector. During the recursive tracing of the CMVP, when a candidate pointed to by the chained motion vector is encoded affinely, the MV of two or three sub-blocks at the corner of that candidate is used to derive the affine model for inheritance. The derived affine model is added to the affine merging or AMVP candidate list for affine merging or AMVP pattern prediction. In some embodiments, if a spatial candidate, non-adjacent candidate, or historical candidate (e.g., stored in 2...) is not encoded affinely, the MV of that ... nd If the base MV in the type affine history parameter table is encoded in an affine manner, then the candidate center MV is set as the initial MV or BV (or base vector) of the chain motion vector prediction.

[0096] B. Affine Construction Candidates

[0097] In some embodiments, affine construction candidates can be derived by recursively tracing MV and BV based on an initial MV or BV. Specifically, the two or three spatial MV or BV at the corner of the CU (denoted as ) , ) is set as the initial MV or BV. During the recursive tracing process of CMVP, when two or three cascading MVs (i.e., , When pointing to the same reference image, an affine construction model can be derived from these cascaded MVs and added to the affine merging or AMVP candidate list for affine merging or AMVP pattern prediction.

[0098] C. Affine Regression Candidates

[0099] In some embodiments, a novel regression-based affine candidate can be derived by recursively tracing the MV and BV based on an initial MV or BV. Specifically, if a spatial candidate or non-adjacent candidate is not encoded affinely, its MV or BV is set as the initial MV or BV for chained motion vector prediction. During the recursive tracing of the CMVP, when the CU pointed to by the cascaded MV is encoded affinely, the affine model is derived using the sub-block MV in the reference CU and the neighboring sub-block MVs of the current CU through a regression-based affine candidate derivation method. The derived affine model is added to the affine merging or AMVP candidate list for affine merging or AMVP pattern prediction.

[0100] In some embodiments, if spatial candidates, non-adjacent candidates, or history-based candidates (e.g., stored in 2...) nd If the base MV in the type affine history parameter table is encoded in an affine manner, then the candidate center MV is set as the initial MV or BV of the CMVP process.

[0101] D. Historically based affine candidates

[0102] In some embodiments, a novel history-based affine candidate can be derived by recursively tracing MV and BV based on an initial MV or BV as a basis vector. Specifically, if spatial candidates, non-adjacent candidates, or history-based candidates (e.g., stored in 2...) nd If the base MVs (Modal Values) in the affine history parameter table are not encoded affinely, then the MVs or BVs of spatial candidates, non-adjacent candidates, or history-based candidates are set as the initial MVs or BVs. During the recursive tracing of CMVP, the cascaded MVs are used as the base MVs, and the positions of the initial MVs or BV candidates are used as base positions. The affine model is derived by obtaining parameters from the affine history parameter table. The derived affine model is added to the affine merging or AMVP candidate list for affine merging or AMVP pattern prediction.

[0103] In some embodiments, if a spatial candidate, a non-adjacent candidate, or a history-based candidate is encoded affinely, then the center MV of that candidate is set as the initial MV or BV of the chained motion vector prediction.

[0104] E. Affine non-adjacent candidate

[0105] In some embodiments, non-adjacent affine candidates can be derived by recursively tracing MV and BV based on an initial MV or BV as a basis vector. Specifically, two or three non-adjacent MVs or BVs at the corner of the CU (i.e., those with the same search pattern as the non-adjacent affine pattern, denoted as ) are selected. , ) is set as the initial MV or BV. During the recursive tracing process of CMVP, when two or three cascading MVs (i.e., , When pointing to the same reference image, a non-adjacent affine model can be derived from these cascaded MVs and added to the affine merging or AMVP candidate list for affine merging or AMVP pattern prediction.

[0106] For the above methods, the CMVP process can be executed after affine inheritance, construction, regression, and construction based on history and non-adjacent candidates.

[0107] In some embodiments, the CU center motion vector (MV) derived from the affine model, or the checkpoint motion vector from affine inheritance, construction, regression, history-based, and non-adjacent candidates, is set as the initial motion vector for chained motion vector prediction. Affine inheritance, regression, and history-based candidates can be derived through the chained motion vector prediction (CMVP) process. When performing chained motion vector prediction using the CU center motion vector, the reference position of the affine model is set as the center of the CU.

[0108] In some embodiments, affine inheritance, construction, regression, history-based, and checkpoint motion vectors of sub-blocks at two or three corners of non-adjacent candidates are set as the initial motion vectors for chained motion vector prediction (denoted as ). , New types of affine constructs and non-adjacent candidates can be derived through the CMVP (i.e., , This is derived from the derivation.

[0109] For the methods described above, if spatial candidates, non-adjacent candidates, or history-based candidates are encoded affinely, the CU center motion vector or checkpoint motion vector derived from the affine model is set as the initial motion vector for chained motion vector prediction. A new type of affine candidate can be derived by inheriting the affine models (i.e., a, b, c, and d) of spatial candidates, non-adjacent candidates, or history-based candidates and using chained motion vectors as the reference motion vector.

[0110] In some embodiments, if the candidates pointed to by the chained motion vectors are encoded affinely, a new affine candidate type can be derived by mixing the affine models pointing to the candidates (i.e., a, b, c, d, e, and f) or checkpoint motion vectors with the affine models or checkpoint motion vectors of the initial spatial candidates, non-adjacent candidates, or history-based candidates. In some embodiments, if the candidates pointed to by the chained motion vectors are encoded affinely, a new affine candidate type can be derived by regression (similar to regression-based affine derivation, with input being a mixture of the sub-block motion vectors pointing to the candidates with the sub-block motion vectors of the initial spatial candidates, non-adjacent candidates, or history-based candidates, minimizing the difference between the mixed sub-block motion vectors and the regression-derived sub-block motion vectors).

[0111] In some embodiments, bidirectional predictive motion vector candidates can be separated into L0 and L1 motion vector candidates during the CMVP process, and the affine model of the current CU can be independently derived from the L0 and L1 chained motion vectors, respectively. In some embodiments, if the sub-block motion vectors or checkpoint motion vectors at two or three corners of the affine candidate are set as the initial motion vectors for chained motion vector prediction, then the tracking motion vectors of these two or three sub-block motion vectors or checkpoint motion vectors point to the same reference image, and new affine candidates can be derived from these tracking motion vectors.

[0112] In some embodiments, if the center motion vector derived from the affine model or the checkpoint motion vector of the affine candidate is set as the initial motion vector for the CMVP process, the checkpoint motion vector of the affine candidate is scaled to point to the reference image pointed to by the tracking motion vector of the center motion vector. New affine candidates can be derived from the tracking motion vector and the scaled checkpoint motion vector.

[0113] V. Sub-block temporal motion vector prediction with chained motion vectors (SbTMVP)

[0114] In some embodiments, additional motion offset candidates for SbTMVP can be generated based on chained motion vector prediction (CMVP). For example, in some embodiments, regular motion offset candidates (e.g., neighboring block A1) can be used as basis vectors to derive chained motion vectors using the CMVP process described in Section III, and the derived chained motion vectors become additional motion offset candidates for SbTMVP.

[0115] The chained motion vectors derived for motion offsets in SbTMVP can be unidirectional or bidirectional predictions. In some embodiments, the derived chained motion vectors can only be used as motion offset candidates for SbTMVP if the corresponding reference image of the unidirectional predicted chained motion vector is a co-located image, or if one of the corresponding reference images of the bidirectional predicted chained motion vector is a co-located image.

[0116] In some embodiments, chained motion vector prediction (CMVP) can be applied to the sub-block motion of a further offset SbTMVP. For example, in some embodiments, the sub-block motion of the SbTMVP is derived by scaling the motion from a co-location image. In this process, the scaling factor is derived based on two POC differences. One is the difference between the POC of the co-location image and the POC of the reference image for the motion of the co-location image, and the other is the difference between the POC of the current image and the POC of the reference image for the current block.

[0117] Figure 13 This conceptually illustrates how sub-block motion is derived by scaling motion from co-located images. As shown, a motion offset based on neighboring block A1 is applied so that the current sub-CU 1315 finds its corresponding co-located sub-CU 1355 in co-located image 1350. The sub-block motion "ScaledrefMv" of the current sub-CU 1315 of the current block in the current image 1310 is derived by scaling the motion "refMv" of co-located sub-CU 1355. The motion vector "refMV" points to a block in reference image 1360. The motion vector "ScaledrefMv" points to a block in reference image 1320. The scaling factor for scaling "refMv" to "ScaledrefMv" is determined by (i) the POC difference "ta" between co-located image 1350 and its reference image 1360 and (ii) the POC difference "tb" between the current image 1310 and its reference image 1320.

[0118] In the figure, the chain motion vector “ChainMV” is derived based on the scaling subblock motion “ScaledrefMv” of the current sub-CU 1315 as its base vector. This chain motion vector is located in a reference block in the reference image 1330 and can be used to generate predictors for encoding or decoding the current sub-CU 1315.

[0119] In some embodiments, the sub-block motion of SbTMVP is a co-location reference motion in a co-location image. That is, the reference block of each sub-block in the co-location image is used to predict the sub-blocks of the current block. Figure 14AThis conceptually illustrates how to predict the current sub-block using a block referenced by a sub-block in the same location image. As shown, a motion offset based on the neighboring block A1 is applied, causing the current sub-CU 1415 to find its corresponding sub-CU 1455 in the same location image 1350. For the current sub-CU 1415 in the current image 1410, its corresponding sub-CU 1455 has a reference motion "refMv" that references block 1465 in the reference image 1460. The reference block 1465 can be used to generate predictors for the sub-CU 1415 of the current block.

[0120] In some embodiments, the sub-block motion of SbTMVP is the motion of a reference block with reference motion at the same location in the same image. That is, the reference block of the reference block of each sub-block in the same image will be used to predict the sub-blocks of the current block. Figure 14B This conceptually illustrates using a reference block of a reference block of a colocation sub-block as a prediction for the current sub-block. In this example, reference block 1465, referenced by colocation sub-CU 1455, has motion that further references reference block 1475 in further reference image 1470. Reference block 1475 can be used to generate a predictor for sub-CU 1415 of the current block. In this example, the motion information of colocation sub-CU 1455 can be referred to as the basis vector of chained MV 1452, which positions reference block 1475 in further reference image 1470.

[0121] In some embodiments, the sub-block motion of SbTMVP is the motion of a reference block in a co-location image with a co-location reference motion (e.g., a chained MV based on co-location sub-blocks). A scaling factor is applied to the chained MV of the co-location sub-blocks to derive the current sub-block motion. This scaling factor is derived based on two POC differences to derive the scaled sub-block motion vector predicted for the current block. One is the difference between the POC of the co-location image and the POC of a reference image, wherein the reference image contains a reference block of the corresponding block in the co-location image. The other is the difference between the POC of the current image and the POC of the reference image of the current block. It should be constrained that the reference image for the scaled sub-block motion vector must be included in the list of reference images for the current block.

[0122] Figure 15This conceptually illustrates scaling a chained motion vector (MV) of a co-location image to the sub-block motion of the current block. As shown, a motion offset based on the neighboring block A1 is applied, causing the current sub-CU 1515 to find its corresponding co-location sub-CU 1555 in the co-location image 1550. The current sub-CU 1515 has a corresponding co-location sub-CU 1555 in the co-location image 1550 based on the motion offset. The co-location sub-CU 1555 has a motion referenced to block 1565 in reference image 1560, and reference block 1565 further references to further reference block 1575 in further reference image 1575, thereby deriving a chained motion vector (MV) 1552 (with the motion of co-location sub-CU 1555 as its basis vector). This chained motion vector 1552 references reference block 1575 from co-location sub-CU 1555 and is then scaled to the sub-block motion 1512 of the current sub-CU 1515. The scaling MV 1512 references reference block 1545 in reference image 1540 to generate predictions for sub-CU 1515. The scaling factor for scaling chained MV 1552 to scaling MV 1512 is calculated based on (i) the POC difference between co-located image 1550 and reference image 1570, and (ii) the POC difference between current image 1510 and reference image 1540.

[0123] In some embodiments, after chained motion vector prediction is applied to the sub-block motion of SbTMVP, a reference block for each derived sub-block motion is used to predict the current block.

[0124] In some embodiments, motion offset candidates are not applied to block-based offsets of SbTMVP candidates, but rather used to offset sub-block motions found in the co-location image. For example, MV(0, 0) is used as the motion offset for the SbTMVP candidate. That is, the sub-block motion of the SbTMVP candidate at the location corresponding to the current block is searched in the co-location image. Then, N sub-block motions may be found and offset together according to the motion vector referenced from the current block's spatial neighbor A1. N is related to the number of sub-blocks in the SbTMVP encoded block. The sub-block motions are then scaled and used to predict the current block. In the above example, the motion used to offset the sub-block motion (MV referenced from spatial neighbor A1 in the above example) can be replaced by other motions referenced from other spatial locations, non-adjacent locations, or from the HMVP table, or by motions derived from chained MV techniques. In some of these embodiments, the motion used to offset the sub-block motion should be a motion from the co-location image. (In the above example, MV(0, 0) can be replaced by any other neighboring MV, non-adjacent MV, HMVP, or chained MV.)

[0125] In some embodiments, the merging information includes a flag indicating whether SbTMVP with sub-block motion offset mode is applied. If this flag is not enabled, conventional SbTMVP is applied. In some embodiments, the reference image for each sub-block on the SbTMVP block is constrained to be the same, but is not limited thereto. In some embodiments, the motion used to offset the sub-block motion should be motion from an image at the same location, but is not limited thereto.

[0126] In some embodiments, an additional list of SbTMVPs is derived, comprising multiple SbTMVPs derived from chained motion vector prediction. In some embodiments, a TM reordering technique may be applied to this additional SbTMVP list. Only the top N candidates with lower TM costs in the list are added to an affine candidate list and compete with the traditional SbTMVPs and affine candidates.

[0127] In some embodiments, an additional list of SbTMVPs is derived, including SbTMVPs derived by chained motion vector prediction and SbTMVPs derived by conventional methods. In some embodiments, if motion is unavailable for a sub-block of an SbTMVP candidate (e.g., the corresponding sub-block is intra-frame or IBC-coded, or the reference image for the sub-block of the SbTMVP derived by chained motion vector prediction is not included in the current list of reference images), the candidate is considered invalid.

[0128] In some embodiments, if motion information for a sub-block of an SbTMVP candidate is unavailable (e.g., the reference image for the corresponding sub-block, derived via intra-frame prediction or IBC coding, or via chained motion vector prediction, is not included in the current reference image list), then the center-derived sub-block motion information will be used. In some embodiments, if motion information for a sub-block of an SbTMVP candidate is unavailable (e.g., the reference image (refPic1) for the SbTMVP sub-block derived via chained motion vector prediction is not included in the current reference image list), then the corresponding reference block will be used to predict the current sub-block from the image whose POC is closest to refPic1 and is in the current reference image list.

[0129] In some embodiments, if motion information for a sub-block of the SbTMVP candidate is unavailable (i.e., the reference image (refPic1) of the sub-block of the SbTMVP derived by chained motion vector prediction is not included in the current reference image list), the motion information will be scaled to the image that is closest to the POC of refPic1 and is in the current reference image list to predict the current sub-block.

[0130] For some embodiments, the above method is not limited to SbTMVP. For example, in some embodiments, chained motion vector prediction can be used in inter-frame merging mode. If the reference image (refPic2) of the derived chained motion vector is not in the current reference image list, the corresponding reference block will be used to predict the current block in the image (targetPic) that is closest to the POC of refPic2 and is in the current reference image list. Furthermore, the corresponding motion information will be scaled to targetPic based on the POC difference between the current image and targetPic. Alternatively, in some embodiments, the corresponding motion information will not be scaled for the current block prediction; instead, it will be directly used to obtain a block in targetPic for the current block prediction.

[0131] In some embodiments, an on / off control flag is transmitted at the coding unit (CU) level, slice level, image level, and / or sequence level to indicate whether the proposed method is enabled. In some embodiments, the proposed method is enabled or disabled based on one or a combination of the selected reference image index, the temporal distance between the reference image and the current image, quantization parameters, coding information of the current CU, prediction mode, motion vector, motion vector accuracy, residual of the current CU, and reference samples.

[0132] VI. Reference picture resampling (RPR)

[0133] In some embodiments, a reference image with a different width or height compared to the current image can be used as a co-location image, and temporal motion vector prediction (TMVP) can be enabled on this reference image. To obtain TMVP on a co-location image with a different width or height, position scaling can be applied to the co-location image.

[0134] Figure 16 This illustrates positional scaling of the same-location image. As shown, to obtain the TMVP at position (x, y) in the current image, the position (x', y') in the same-location image should be used. The position (x', y') is derived based on the width and height ratio of the current image to the same-location image. For example, to obtain the TMVP at position (64, 64) in the current image with a width of 1920 and a height of 1080, the corresponding position in the same-location image with a width of 3840 and a height of 2160 is (128, 128).

[0135] In some embodiments, for SbTMVP cases on co-located images with different widths or heights, a position scaling method can be applied after the motion offset. For example, to obtain the sub-block MVP at position (64, 64) in the current image with a width of 1920 and a height of 1080, the derived motion offset is (4, 4). In one case, in co-located images with the same width and height, the corresponding position is (68, 68). In another case, in co-located images with a width of 3840 and a height of 2160, the corresponding position is (136, 136).

[0136] In some embodiments, position scaling can be applied during CMVP derivation to handle differences in the width or height of reference images. Position scaling should be applied whenever two adjacent reference images in the chain differ in width or height. In some embodiments, position scaling can be applied to the process of acquiring any information from other images with different widths or heights.

[0137] See Figure 11 For example, if the reference image RefPicL0 k(1) The width or height is the same as the reference image RefPicL0 k(0) If they are different, then the position of the next motion vector component, RefPicL0, is used to derive it. k(1) (i.e., MVL0) k(2) ) ) should be based on RefPicL0 k(1) and RefPicL0 k(0) The width and height are scaled proportionally.

[0138] In some embodiments, if the current POC is similar to a reference POC (e.g., RefPicL0) k(0) The difference between the current POC and the POC of the reference image pointed to by the MVP candidate (e.g., RefPicL0) is used. k(1) If the difference signs between ). ) are the same, then the MV candidate is considered for chained MVP construction (or given priority in the existing list). This means that CMVPs that do not change the prediction direction from forward to backward or from backward to forward should be used for construction or given priority in the existing list.

[0139] In some embodiments, when the MVP candidate (MVP) k(1) When the process is bidirectional, the MVP pointing in the same direction is selected between the two to accumulate MVP construction (e.g., MVL0). k(1) When both MVPs point in the same direction as the prediction direction between the current image and the reference image, the MVP pointing to the POC that is closest to the reference image (e.g., RefPicL0) should be selected. k(0)) ).

[0140] The proposed method can be implemented in the encoder and / or decoder. For example, the proposed method can be implemented in the loop filtering module of the encoder and / or the loop filtering module of the decoder.

[0141] VII. Example Video Codec

[0142] Figure 17 An example video codec 1700 capable of implementing sub-block motion is shown. As shown, the video codec 1700 receives an input video signal from a video source 1705 and encodes the signal into a bitstream 1795. The video codec 1700 has multiple components or modules for encoding the signal from the video source 1705, including at least some components selected from the transform module 1710, quantization module 1711, inverse quantization module 1714, inverse transform module 1715, intra-frame estimation module 1724, intra-frame prediction module 1725, motion compensation module 1730, motion estimation module 1735, loop filter 1745, reconstructed image buffer 1750, motion vector buffer 1765, motion vector prediction module 1775, and entropy encoder 1790. The motion compensation module 1730 and motion estimation module 1735 belong to the inter-frame prediction module 1740. Intra-frame prediction module 1725 and intra-frame prediction estimation module 1724 belong to current image prediction module 1720. Current image prediction module 1720 uses current image reconstruction samples as reference samples for predicting the current block.

[0143] In some embodiments, modules 1710–1790 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 1710–1790 are hardware circuit modules implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 1710–1790 are illustrated as independent modules, some of these modules may be combined into a single module.

[0144] Video source 1705 provides an uncompressed video signal, presenting pixel data for each video frame. Subtractor 1708 calculates the difference between the raw video pixel data from video source 1705 and the predicted pixel data 1713 from motion compensation module 1730 or intra-frame prediction module 1725, as a prediction residual 1709. Transform module 1710 converts this difference (or residual pixel data or residual signal 1708) into transform coefficients (e.g., by performing a Discrete Cosine Transform, or DCT for short). Quantization module 1711 quantizes the transform coefficients into quantized data (or quantization coefficients) 1712, which is encoded into a bitstream 1795 by entropy encoder 1790.

[0145] The inverse quantization module 1714 inverse-quantizes the quantized data (or quantization coefficients) 1712 to obtain transform coefficients 1718. The inverse transform module 1715 performs an inverse transform on the transform coefficients 1718 to generate a reconstruction residual 1719. The reconstruction residual 1719 is added to the predicted pixel data 1713 to generate reconstructed pixel data 1717. In some embodiments, the reconstructed pixel data 1717 is temporarily stored in the line buffer 1727 (or intra-frame prediction buffer) for intra-frame prediction and spatial motion vector (MV) prediction. The reconstructed pixels are filtered by the loop filter 1745 and then stored in the reconstructed image buffer 1750. In some embodiments, the reconstructed image buffer 1750 is external memory to the video codec 1700. In some embodiments, the reconstructed image buffer 1750 is internal memory to the video codec 1700.

[0146] Intra-frame estimation module 1724 performs intra-frame prediction based on reconstructed pixel data 1717 to generate intra-frame prediction data. The intra-frame prediction data is provided to entropy encoder 1790 for encoding into a bitstream 1795. The intra-frame prediction data is also used by intra-frame prediction module 1725 to generate predicted pixel data 1713.

[0147] The motion estimation module 1735 performs inter-frame prediction of reference pixel data from previously decoded frames stored in the reconstructed image buffer 1750 by generating motion vectors (MVs). These motion vectors are then provided to the motion compensation module 1730 to generate predicted pixel data.

[0148] Instead of encoding the complete actual motion vectors into the bitstream, the video codec 1700 uses motion vector prediction to generate predicted motion vectors and encodes the difference between the motion vectors used for motion compensation and the predicted motion vectors as residual motion data and stores it in the bitstream 1795.

[0149] The motion vector prediction module 1775 generates predicted motion vectors based on reference motion vectors generated for encoding previous video frames (i.e., motion-compensated motion vectors used to perform motion compensation). The motion vector prediction module 1775 retrieves the reference motion vectors from the motion vector buffer 1765 for the previous video frames. The video codec 1700 stores the motion vectors generated for the current video frame in the motion vector buffer 1765 as reference motion vectors for generating the predicted motion vectors.

[0150] The motion vector prediction module 1775 generates predicted motion vectors using reference motion vectors. The predicted motion vectors can be calculated through spatial motion vector prediction or temporal motion vector prediction. The difference (residual motion data) between the predicted motion vector and the motion-compensated motion vector (MC MV) of the current frame is encoded into a bitstream 1795 by the entropy encoder 1790.

[0151] The entropy encoder 1790 encodes various parameters and data into a bitstream 1795 using entropy coding techniques such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding. The entropy encoder 1790 encodes various header elements, flags, quantization transform coefficients 1712, and residual motion data as syntax elements into the bitstream 1795. The bitstream 1795 is then stored in a storage device or transmitted to the decoder via a communication medium such as a network.

[0152] The loop filter 1745 performs filtering or smoothing operations on the reconstructed pixel data 1717 to reduce coded artifacts, particularly at pixel block boundaries. In some embodiments, the filtering or smoothing operations performed by the loop filter 1745 include a deblock filter (DBF), a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF). In some embodiments, luma mapping chroma scaling (LMCS) is performed before the loop filtering.

[0153] Figure 18The diagram illustrates a portion of the video codec 1700 that implements sub-block processing and concatenated vectors. As shown, the sub-block processing module 1810 receives a motion offset indication, which is selected by the entropy encoder 1790 from a motion offset candidate list 1820. The motion offset candidates 1820 include neighboring blocks of the current block (e.g., A1), and concatenated vectors derived from the motion information of these neighboring blocks as basis vectors. The motion information of the neighboring blocks is provided by a motion vector buffer 1765, which also provides motion vectors and block vectors (BVs) to the vector concatenator 1830 to generate the concatenated vectors.

[0154] The sub-block processing module 1810 performs several functions: co-location sub-block recognition 1811, vector concatenation 1812, time scaling 1813, and position scaling 1814. The co-location sub-block recognition function 1811 uses a selected motion offset to identify the co-location image and the co-location sub-blocks corresponding to the current sub-block. The vector concatenation function 1812 is based on certain basis vectors (which can be motion information of the co-location sub-blocks, or a combination thereof, as described above). Figure 13-15 The motion information of the sub-blocks at the same position in the current image is time-scaled to generate a cascaded vector (or chained motion vector). The time scaling function 1813 can time-scale the motion information based on a scaling factor calculated from: (i) the time difference between the current image and its sub-block reference image, and (ii) the time difference between the image at the same position and its sub-block reference image. The position scaling function 1814 can position-scale the image at the same position if the height / width of the image at the same position is different from the current image.

[0155] The sub-block processing module 1810 outputs the motion field for the current sub-block (i.e., a sub-block of the current block) to the inter-frame prediction module 1740. This motion field is determined by sub-block recognition 1811, vector concatenation 1812, and temporal scaling 1813. The inter-frame prediction module 1740 obtains samples from the reconstructed image buffer 1750 and the line buffer 1727 based on the motion field to generate prediction sub-blocks for the current block, which become prediction pixel data 1713.

[0156] Figure 19 The process 1900 for encoding sub-blocks is conceptually illustrated. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing encoder 1700 execute process 1900 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing encoder 1700 executes process 1900.

[0157] The encoder receives the data to be encoded in block 1910, which serves as the current pixel block of the current video image. This current block comprises multiple current sub-blocks.

[0158] The encoder identifies multiple co-location sub-blocks in the co-location image corresponding to the multiple current sub-blocks based on motion offsets in block 1920. In some embodiments, when the width or height of the co-location image differs from the current image, the encoder applies position scaling to the co-location image. In some embodiments, the motion offset is provided by a cascaded vector (or chained motion vector, abbreviated as MV), which is the sum of at least two recursive tracking vectors, each of which is a motion vector or a block vector. The cascaded vector has a base vector, which is the motion vector or block vector of the neighboring block of the current block. In some embodiments, the motion offset candidate list includes: (i) the block vector or motion vector of the neighboring block, which is the base vector of the cascaded vector, and (ii) the cascaded vector.

[0159] In block 1930, the encoder derives predictors for the plurality of current sub-blocks based on the motion information of the plurality of co-located sub-blocks. In some embodiments, the predictors for the plurality of current sub-blocks are derived by scaling the motion information of the plurality of co-located sub-blocks according to the time distance between the current image, the co-located image, and a reference image of the motion information of the plurality of co-located sub-blocks.

[0160] In some embodiments, a cascaded vector (or chained MV) is derived based on the motion information of the sub-blocks at the same location to derive the predictor of the current sub-block. The cascaded vector is the sum of at least two recursive tracking vectors, each of which is a motion vector or a block vector.

[0161] In some embodiments, as described above Figure 13 The motion information of the co-located sub-block is time-scaled to the current image to be used as a basis vector for deriving the cascaded vector and identifying a reference block from the current sub-block, wherein the scaling is applied according to a scaling factor based on: (i) a first time difference between the co-located image and the reference image identified by the motion information of the co-located sub-block; and (ii) a second time difference between the current image and the reference image identified by the cascaded vector.

[0162] In some embodiments, the motion information of the co-located sub-block is used as the basis vector for deriving the cascade vector. (As in combination) Figure 14B The predictor for the current sub-block can be derived based on a reference block identified from the co-located sub-block by the concatenated vector. In some embodiments, as described above... Figure 15The predictor of the current sub-block can be derived from a reference block identified from the current sub-block by a scaling motion vector, which is derived by temporally scaling the concatenated vector to the current image, wherein the scaling is applied according to a scaling factor based on: (i) a first time difference between the co-located image and the reference image identified by the concatenated vector; and (ii) a second time difference between the current image and the reference image identified by the scaling motion vector.

[0163] In block 1940, the encoder uses the derived predictors of the multiple current sub-blocks to encode the current block in order to generate prediction residuals.

[0164] VIII. Example Video Decoder

[0165] In some embodiments, the encoder may signal (or generate) one or more syntax elements in the bitstream so that the decoder can parse the one or more syntax elements from the bitstream.

[0166] Figure 20 An example video decoder 2000 capable of implementing sub-block motion is illustrated. As shown, the video decoder 2000 is an image or video decoding circuit that receives a bitstream 2095 and decodes its contents into pixel data for video frames to be displayed. The video decoder 2000 has multiple components or modules for decoding the bitstream 2095, including a subset of components selected from an inverse quantization module 2014, an inverse transform module 2015, an intra-frame prediction module 2025, a motion compensation module 2030, a loop filter 2045, a decoded image buffer 2050, a motion vector buffer 2065, a motion vector prediction module 2075, and a resolver 2090. The motion compensation module 2030 is part of the inter-frame prediction module 2040. The intra-frame prediction module 2025 is part of the current image prediction module 2020, which uses reconstructed samples of the current image as reference samples for predicting the current block.

[0167] In some embodiments, modules 2014–2090 are software instruction modules executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 2014–2090 are hardware circuit modules implemented by one or more integrated circuits of an electronic device. Although modules 2014–2090 are illustrated as independent modules, some modules may be combined into a single module.

[0168] Parser 2090 (or entropy decoder) receives bitstream 2095 and performs initial parsing according to the syntax defined by the video coding or image coding standard. The parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 2012. Parser 2090 parses the various syntax elements using entropy coding techniques, such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding.

[0169] The inverse quantization module 2014 inverse-quantizes the quantized data (or quantization coefficients) 2012 to obtain transform coefficients. The inverse transform module 2015 inverse-transforms the transform coefficients 2018 to generate the reconstructed residual signal 2019. The reconstructed residual signal 2019 is added to the predicted pixel data 2013 from the intra-frame prediction module 2025 or the motion compensation module 2030 to generate the decoded pixel data 2017. The decoded pixel data is filtered by the loop filter 2045 and stored in the decoded image buffer 2050. In some embodiments, the decoded image buffer 2050 is an external memory of the video decoder 2000. In some embodiments, the decoded image buffer 2050 is an internal memory of the video decoder 2000.

[0170] Intra-prediction module 2025 receives intra-prediction data from bitstream 2095 and generates predicted pixel data 2013 from decoded pixel data 2017 stored in decoded image buffer 2050 accordingly. In some embodiments, decoded pixel data 2017 is also stored in line buffer 2027 (or intra-prediction buffer) for intra-image prediction and spatial motion vector prediction.

[0171] In some embodiments, the contents of the decoded image buffer 2050 are used for display. The display device 2005 can directly retrieve the contents from the decoded image buffer 2050 for display, or retrieve the contents of the decoded image buffer into a display buffer. In some embodiments, the display device receives pixel values ​​from the decoded image buffer 2050 via pixel transfer.

[0172] The motion compensation module 2030 generates predicted pixel data 2013 from the decoded pixel data 2017 stored in the decoded image buffer 2050 based on motion compensation motion vectors (MC MV). These motion compensation motion vectors are decoded by adding the residual motion data received from the bitstream 2095 to the predicted motion vectors received from the motion vector prediction module 2075.

[0173] The motion vector prediction module 2075 generates predicted motion vectors based on reference motion vectors generated for decoding previous video frames, such as motion-compensated motion vectors used for performing motion compensation. The motion vector prediction module 2075 retrieves reference motion vectors from the motion vector buffer 2065 for the previous video frames. The video decoder 2000 stores the motion-compensated motion vectors generated for decoding the current video frame in the motion vector buffer 2065 as reference motion vectors for generating the predicted motion vectors.

[0174] The in-loop filter 2045 performs filtering or smoothing operations on the decoded pixel data 2017 to reduce coding artifacts, particularly at pixel block boundaries. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 2045 include deblocking filtering (DBF), sample adaptive offset (SAO), and / or adaptive loop filtering (ALF). In some embodiments, luma mapping chroma scaling (LMCS) is performed before loop filtering.

[0175] Figure 21 The diagram illustrates a portion of the video decoder 2000 that implements sub-block processing and concatenated vectors. As shown, the sub-block processing module 2110 receives a motion offset indication, which is selected by the entropy decoder 2090 from a motion offset candidate list 2120. The motion offset candidates 2120 include neighboring blocks of the current block (e.g., A1), and concatenated vectors derived from the motion information of these neighboring blocks as base vectors. The motion information of the neighboring blocks is provided by a motion vector buffer 2065, which also provides motion vectors and block vectors to a vector concatenator 2130 to generate the concatenated vectors.

[0176] The sub-block processing module 2110 performs several functions: co-location sub-block recognition 2111, vector concatenation 2112, temporal scaling 2113, and positional scaling 2114. The co-location sub-block recognition function 2111 uses a selected motion offset to identify co-location images and co-location sub-blocks corresponding to the current block's sub-blocks. The vector concatenation function 2112 is based on certain basis vectors (which can be motion information of co-location sub-blocks, or a combination thereof, as described above). Figure 13-15The motion information of the sub-blocks at the same position in the current image is used to create a cascaded vector (or chained motion vector). The time scaling function 2113 can time-scale the motion information based on a scaling factor calculated based on (i) the time difference between the current image and its reference images, and (ii) the time difference between the image at the same position and its reference images. The position scaling function 2114 can position-scale the image at the same position if its height / width differs from the current image.

[0177] The sub-block processing module 2110 outputs the motion field of the current sub-block (i.e., the sub-block of the current block) to the inter-frame prediction module 2040. This motion field is determined by sub-block recognition 2111, vector concatenation 2112, and temporal scaling 2113. The inter-frame prediction module 2040 extracts samples from the decoded image buffer 2050 and line buffer 2027 based on the motion field to generate predictors for the current block's sub-blocks, thereby obtaining predicted pixel data 2013.

[0178] Figure 22 A conceptual illustration of a flow 2200 for decoding concatenated vectors of sub-blocks is provided. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the decoder 2000 execute flow 2200 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the decoder 2000 executes flow 2200.

[0179] In step 2210, the decoder receives the data to be decoded as the current pixel block of the current video image. This current block includes multiple current sub-blocks.

[0180] In step 2220, the decoder identifies multiple co-location sub-blocks in the co-location image corresponding to multiple current sub-blocks based on motion offsets. In some embodiments, when the width or height of the co-location image differs from the current image, the decoder applies positional scaling to the co-location image. In some embodiments, the motion offset is provided by a cascaded vector (or chained motion vector), which is the sum of at least two recursive tracking vectors, each of which is a motion vector or block vector. The cascaded vector has a base vector, which is the motion vector or block vector of the neighboring blocks of the current block. In some embodiments, the motion offset candidate list includes: (i) the block vectors or motion vectors of the neighboring blocks that serve as the base vector of the cascaded vector, and (ii) the cascaded vector.

[0181] In step 2230, the decoder derives predictors for multiple current sub-blocks based on motion information from multiple co-located sub-blocks. In some embodiments, the predictors for multiple current sub-blocks are derived by scaling the motion information of multiple co-located sub-blocks according to the time distance between the current image, the co-located image, and a reference image of the motion information of multiple co-located sub-blocks.

[0182] In some embodiments, a cascaded vector (or chained motion vector) is derived based on the motion information of the sub-blocks at the same location to derive the predictor of the current sub-block. The cascaded vector is the sum of at least two recursive tracking vectors, each of which is a motion vector or a block vector.

[0183] In some embodiments, as described above Figure 13 The motion information of the co-located sub-blocks is time-scaled to the current image to be used as the basis vector for deriving the cascaded vectors and identifying the reference block from the current sub-block. The scaling is applied according to a scaling factor based on: (i) a first time difference between the co-located image and the reference image identified by the motion information of the co-located sub-blocks; and (ii) a second time difference between the current image and the reference image identified by the cascaded vectors.

[0184] In some embodiments, the motion information of co-located sub-blocks is used as the basis vectors for deriving cascade vectors. (For example, in combination with...) Figure 14B The predictor of the current sub-block can be derived based on a reference block identified from the co-located sub-block by a concatenated vector. In some embodiments, the predictor of the current sub-block can be derived based on a reference block identified from the current sub-block by a scaled motion vector, which is obtained by time-scaling the concatenated vector to the current image, as described above. Figure 15 The scaling is applied based on a scaling factor, which is calculated based on: (i) a first time difference between the co-located image and a reference image identified by the cascaded vector; and (ii) a second time difference between the current image and a reference image identified by the scaling motion vector.

[0185] The decoder reconstructs the current block at block 2240 by utilizing the predicted sub-blocks and corresponding prediction residuals derived from multiple current sub-blocks. The decoder can then display the reconstructed current block as part of the reconstructed current image.

[0186] IX. Example Electronic Systems

[0187] Many of the features and applications described above are implemented as software processes, which are specified as a set of instructions recorded on a computer-readable storage medium (also known as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), the processing unit performs the operations indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard disks, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. Computer-readable media do not include carrier waves and electronic signals transmitted via wireless or wired connections.

[0188] In this specification, the term "software" includes firmware residing in read-only memory or application programs stored in magnetic memory, which can be read into memory and processed by a processor. Furthermore, in some embodiments, multiple software inventions may be implemented as sub-parts of a larger program while still remaining distinct software inventions. In some embodiments, multiple software inventions may also be implemented as independent programs. Finally, any combination of independent programs that together implement the software inventions described herein falls within the scope of this disclosure. In some embodiments, when a software program is installed to run on one or more electronic systems, one or more specific machine implementations are defined for executing and implementing the operations of the software program.

[0189] Figure 23 An electronic system 2300 that may implement certain embodiments of the present disclosure is conceptually illustrated. The electronic system 2300 may be a computer (e.g., a desktop computer, personal computer, tablet computer, etc.), a telephone, a personal digital assistant (PDA), or any other type of electronic device. Such electronic systems include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 2300 includes a bus 2305, a processing unit 2310, a graphics-processing unit (GPU) 2315, system memory 2320, a network 2325, read-only memory 2330, permanent storage device 2335, input device 2340, and output device 2345.

[0190] Bus 2305 collectively represents all system, peripheral, and chipset buses used for communication connections among the numerous internal devices of electronic system 2300. For example, bus 2305 communicates with processing unit 2310, GPU 2315, read-only memory 2330, system memory 2320, and permanent storage device 2335.

[0191] Processing unit 2310 retrieves instructions for execution and data for processing from these different storage units, thereby executing the processes of this disclosure. The processing unit may be a single processor or a multi-core processor in different embodiments. Some instructions are passed to GPU 2315 and executed thereon. GPU 2315 may offload various computational tasks or supplement the image processing provided by processing unit 2310.

[0192] Read-only memory (ROM) 2330 stores static data and instructions used by processing unit 2310 and other modules of the electronic system. Permanent storage device 2335 is a read-write storage device. This device is a non-volatile storage unit that can store instructions and data even when the electronic system 2300 is powered off. Some embodiments of this disclosure use mass storage devices (such as disks or optical discs and their corresponding disk drives) as permanent storage device 2335.

[0193] Other embodiments use removable storage devices (such as floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Similar to permanent storage device 2335, system memory 2320 is also a read-write storage device. However, unlike storage device 2335, system memory 2320 is volatile read-write memory, such as random access memory. System memory 2320 stores some instructions and data used by the processor during runtime. In some embodiments, processes conforming to this disclosure are stored in system memory 2320, permanent storage device 2335, and / or read-only memory 2330. For example, various storage units include instructions for processing multimedia segments, conforming to some embodiments. From these different storage units, processing unit 2310 retrieves instructions for execution and data for processing, thereby executing the processes of some embodiments.

[0194] Bus 2305 also connects input device 2340 and output device 2345. Input device 2340 enables a user to pass information to the electronic system and select commands. Input device 2340 includes an alphanumeric keypad and positioning device (also known as a "cursor control device"), a camera (e.g., a webcam), a microphone, or similar devices for receiving voice commands. Output device 2345 displays images generated by the electronic system or otherwise outputs data. Output device 2345 includes printers and display devices such as cathode ray tube (CRT) or liquid crystal display (LCD), and speakers or similar audio output devices. Some embodiments include devices such as touchscreens that can function as both input and output devices.

[0195] Finally, as Figure 23As shown, bus 2305 also connects electronic system 2300 to network 2325 via a network adapter (not shown). In this way, the computer can become part of a computer network (such as a local area network (LAN), wide area network (WAN), or intranet, or a network of networks such as the Internet). Any or all components of electronic system 2300 can be used in conjunction with this disclosure.

[0196] Some embodiments include electronic components such as microprocessors, memory, and storage devices for storing computer program instructions in a machine-readable or computer-readable medium (also referred to as a computer-readable storage medium, machine-readable medium, or machine-readable storage medium). Examples of such computer-readable media include random access memory (RAM), read-only memory (ROM), read-only optical disc (CD-ROM), recordable optical disc (CD-R), rewritable optical disc (CD-RW), read-only digital versatile optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state drives, read-only and recordable Blu-ray discs, ultra-high-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable medium may store a computer program executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code generated by a compiler, and files containing high-level code that are executed by a computer, electronic component, or microprocessor via an interpreter.

[0197] While the foregoing discussion primarily concerns microprocessors or multi-core processors executing software, many of the aforementioned features and applications are executed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself. Furthermore, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

[0198] In this specification and any claim of this application, the terms "computer," "server," "processor," and "memory" refer to electronic or other technical devices. These terms do not include individuals or groups of individuals. For the purposes of this specification, the term "display" or "display operation" means displaying on an electronic device. In this specification and any claim of this application, the terms "computer-readable medium," "computer-readable media," and "machine-readable medium" are fully defined as tangible, physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, or any other transient signals.

[0199] Although this disclosure has been described in conjunction with numerous specific details, those skilled in the art will recognize that this disclosure may be implemented in other specific forms without departing from its spirit. Furthermore, several accompanying drawings (including...) Figure 19 and Figure 22 This provides a conceptual description of the process. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous sequence, and different specific operations may be performed in different embodiments. Furthermore, the process may be implemented through several sub-processes or as part of a larger macro-process. Therefore, those skilled in the art should understand that this disclosure should not be limited to the foregoing illustrative details but should be defined by the appended claims.

[0200] Additional notes

[0201] The topics described herein sometimes schematically illustrate different components contained within or connected to different other components. It should be understood that such illustrated architectures are merely examples, and many other architectures can actually be implemented to achieve the same functionality. Conceptually, any arrangement of components to achieve the same functionality is considered "associated" to achieve the desired function. Therefore, any two components combined in this document to achieve a specific function can be considered "associated" to achieve the desired function, regardless of the architecture or intermediate components. Similarly, any two such associated components can also be considered "operationally connected" or "operationally coupled" to achieve the desired function, and any two components that can be suchly associated can also be considered "operationally coupled" to achieve the desired function. Specific examples of operational coupling include, but are not limited to, physically matable and / or physically interactive components and / or wirelessly interactive components and / or logically interactive and / or logically interactive components.

[0202] Furthermore, regarding the use of almost all plural and / or singular terms in this document, those skilled in the art can convert plural to singular and / or singular to plural as needed by the context and / or application. For clarity, various singular / plural permutations may be explicitly listed herein.

[0203] Furthermore, those skilled in the art will understand that, in general, the terms used herein, particularly in the appended claims, such as the body portion of the appended claims, are typically intended as “open-ended” terms. For example, the word “comprising” should be interpreted as “including but not limited to,” the word “having” should be interpreted as “having at least,” and the word “including” should be interpreted as “including but not limited to,” etc. Those skilled in the art will also further understand that if a particular quantity introduced in a claim is intentionally limited, that intention will be explicitly stated in the claim; if no such statement is made, that intention does not exist. For example, for ease of understanding, the appended claims below may contain the introductory phrases “at least one” and “one or more” to introduce claim elements. However, the use of such phrases should not be construed as meaning that a claim element introduced by the indefinite article “a” or “an” will limit any particular claim containing that element to containing only one of that element, even if the same claim contains both the introductory phrase “one or more” or “at least one” and the indefinite article “a” or “an,” for example, “a” and / or “an” should be interpreted as “at least one” or “one or more”; the same applies to definite articles used to introduce claim elements. Furthermore, even if a specific number of elements is explicitly stated in the claims, those skilled in the art will recognize that such a statement should be interpreted as at least the stated number. For example, the statement "two elements" alone, without other modifiers, means at least two elements, or two or more elements. Additionally, when expressions such as "at least one A, B, and C" are used, they should generally be interpreted according to the meaning understood by those skilled in the art. For example, "a system having at least one A, B, and C" should include, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and systems having both A, B, and C. Similarly, when expressions such as "at least one A, B, or C" are used, they should generally be interpreted according to the meaning understood by those skilled in the art. For example, "a system having at least one A, B, or C" should include, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and systems having both A, B, and C. Those skilled in the art will further understand that almost all extractive terms and / or phrases presenting two or more alternative terms in the specification, claims, or drawings should be understood to cover all possibilities including one term, any term, or both terms. For example, the term "A or B" should be understood to include all possibilities including "A" or "B" or "A and B".

[0204] As can be seen from the foregoing, various embodiments of this disclosure have been described for illustrative purposes, and various modifications can be made without departing from the scope and spirit of this disclosure. Therefore, the various embodiments disclosed herein are not intended to be limiting, but rather to be understood within the true scope and spirit indicated by the following claims.

Claims

1. A video encoding / decoding method, comprising: Receive data to be encoded or decoded as the current pixel block of the current video image, wherein the current block includes multiple current sub-blocks; Based on the motion offset provided by the cascaded vector, which is the sum of at least two recursive tracking vectors, each tracking vector being a motion vector or a block vector, multiple co-located sub-blocks in the co-located image corresponding to the multiple current sub-blocks are identified; Based on the motion information of the multiple co-located sub-blocks, the prediction sub-blocks of the multiple current sub-blocks are derived; as well as The current block is encoded or decoded using the predicted sub-blocks derived from the multiple current sub-blocks.

2. The video encoding / decoding method according to claim 1, wherein the concatenated vector has a base vector, and the base vector is the motion vector or block vector of the neighboring blocks of the current block.

3. The video encoding / decoding method according to claim 2, wherein the motion offset candidate list includes: (i) the block vector or motion vector of the neighboring block, which is the base vector of the cascade vector, and (ii) the cascade vector.

4. The video encoding and decoding method according to claim 1, wherein the motion information of the plurality of co-located sub-blocks is scaled according to the time distance between the current image, the co-located image and the reference image of the motion information of the plurality of co-located sub-blocks, and the prediction sub-block of the plurality of current sub-blocks is derived.

5. The video encoding / decoding method according to claim 1, further comprising: When the width or height of the image at the same location is different from the current image, position scaling is applied to the image at the same location.

6. A video encoding / decoding method, comprising: Receive data to be encoded or decoded as the current pixel block of the current video image, wherein the current block includes multiple current sub-blocks; Based on motion offset, identify multiple co-located sub-blocks in the co-located image corresponding to the multiple current sub-blocks; Based on the motion information of the multiple co-located sub-blocks, predictors of the multiple current sub-blocks are derived, wherein a concatenated vector is derived based on the motion information of the co-located sub-blocks to derive the predictor of the current sub-block, and the concatenated vector is the sum of at least two recursive tracking vectors, each tracking vector being a motion vector or a block vector; as well as The current block is encoded or decoded using the predicted sub-blocks derived from the multiple current sub-blocks.

7. The video encoding / decoding method according to claim 6, wherein the motion information of the co-located sub-blocks is used as the basis vector for deriving the cascaded vector.

8. The video encoding / decoding method of claim 7, wherein the predictor of the current sub-block is derived based on a reference block identified from the co-located sub-block by the concatenated vector.

9. The video encoding / decoding method of claim 7, wherein the predictor of the current sub-block is derived from a reference block identified from the current sub-block by a time-scaled motion vector, the time-scaled motion vector being derived by time-scaling the concatenated vector to the current image.

10. The video encoding / decoding method of claim 9, wherein the scaling is applied according to a scaling factor, the scaling factor being calculated based on: (i) a first time difference between the co-located image and a reference image identified by the concatenated vector; and (ii) a second time difference between the current image and a reference image identified by the scaling motion vector.

11. The video encoding / decoding method of claim 6, wherein the motion information of the co-located sub-blocks is time-scaled to the current image to be used as a basis vector for deriving the cascaded vectors and identifying a reference block from the current sub-block.

12. The video encoding / decoding method of claim 11, wherein the scaling is applied according to a scaling factor based on: (i) a first time difference between the co-located image and a reference image identified by the motion information of the co-located sub-block; and (ii) a second time difference between the current image and a reference image identified by the concatenated vector.

13. An electronic device, comprising: Video codec circuitry configured to perform the following operations: Receive data to be encoded or decoded as the current pixel block of the current video image, wherein the current block includes multiple current sub-blocks; Based on motion offset, identify multiple co-located sub-blocks in the co-located image corresponding to the multiple current sub-blocks; Based on the motion information of the multiple co-located sub-blocks, predictors of the multiple current sub-blocks are derived, wherein a concatenated vector is derived based on the motion information of the co-located sub-blocks to derive the predictor of the current sub-block, and the concatenated vector is the sum of at least two recursive tracking vectors, each tracking vector being a motion vector or a block vector; as well as The current block is encoded or decoded using the predicted sub-blocks derived from the multiple current sub-blocks.

14. A video decoding method, comprising: The data to be decoded is received as the current pixel block of the current image of the video, and the current block includes multiple current sub-blocks; Based on motion offset, identify multiple co-located sub-blocks in the co-located image corresponding to the multiple current sub-blocks; Based on the motion information of the multiple co-located sub-blocks, predictors of the multiple current sub-blocks are derived, wherein a concatenated vector is derived based on the motion information of the co-located sub-blocks to derive the predictor of the current sub-block, and the concatenated vector is the sum of at least two recursive tracking vectors, each tracking vector being a motion vector or a block vector; as well as The current block is reconstructed using the predicted sub-blocks derived from the multiple current sub-blocks.

15. A video coding method, comprising: Receive the data to be encoded as the current pixel block of the current image of the video, wherein the current block includes multiple current sub-blocks; Based on motion offset, identify multiple co-located sub-blocks in the co-located image corresponding to the multiple current sub-blocks; Based on the motion information of the multiple co-located sub-blocks, predictors are derived for the multiple current sub-blocks. A concatenated vector is derived based on the motion information of the co-located sub-blocks to derive the predictor of the current sub-block. The concatenated vector is the sum of at least two recursive tracking vectors, each of which is a motion vector or a block vector. as well as The current block is encoded by using the predicted sub-blocks derived from the plurality of current sub-blocks.

Images