Method, device, storage medium, program product and storage method for coding a video bitstream

Abstract

The present disclosure provides methods, devices and non-transitory storage media for decoding a video bitstream. One or more vector predictors associated with a current block can be retrieved from a reference motion vector candidate library, the one or more retrieved motion vector predictors including at least one or more motion vectors associated with one or more decoded blocks, and the one or more decoded blocks belonging to a same superblock as the current block. A motion vector associated with the current block is determined based on the one or more retrieved motion vector predictors, and the current block is decoded based on the determined motion vector. The reference motion vector candidate library can be updated by inserting the motion vector associated with the current block into the reference motion vector candidate library.

Description

[0001] Cross-reference to related applications

[0002] This disclosure claims priority to U.S. Provisional Patent Application No. 63 / 342,744, filed May 17, 2022, with the United States Patent and Trademark Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] Embodiments of this disclosure relate to image and video coding techniques. More specifically, embodiments of this disclosure relate to improvements in generating and parsing a library of reference motion vector candidates. Background Technology

[0004] AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. It was developed by the Open Media Consortium (AOMedia) as a successor to VP9, ​​an alliance founded in 2015 that includes semiconductor companies, video-on-demand providers, video content producers, software development companies, and web browser vendors. Many components of the AV1 project stem from previous research by consortium members. Individual contributors began experimenting with technology platforms several years ago: Xiph / Mozilla's Daala released code in 2010, Google's experimental VP9 evolution project VP10 was announced on September 12, 2014, and Cisco's Thor released it on August 11, 2015. Building upon the VP9 codebase, AV1 incorporates other technologies, some of which were developed in these experimental forms. The first version of the AV1 reference codec (version 0.1.0) was released on April 7, 2016. The consortium announced the release of the AV1 bitstream specification, along with reference software-based encoders and decoders, on March 28, 2018. On June 25, 2018, a confirmed version 1.0.0 of the specification was released. On January 8, 2019, a confirmed version 1.0.0 of the specification, including Errata Table 1, was released. The AV1 stream specification includes a reference video codec.

[0005] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) released the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Since then, they have been studying the potential need for standardization of future video coding technologies that significantly outperform HEVC in compression capabilities. In October 2017, they released a Joint Proposal (CfP) on Video Compression Capabilities Beyond HEVC. As of February 15, 2018, a total of 22 CfP responses had been submitted regarding Standard Dynamic Range (SDR), 12 regarding High Dynamic Range (HDR), and 12 regarding 360 video categories. In April 2018, at the 122MPEG / 10th JVET (Joint Video Exploration Group - Joint Video Experts Group) meeting, all CfP feedback received was evaluated. Following careful evaluation, JVET officially launched the standardization of the next-generation video coding standard beyond HEVC, namely the so-called Universal Video Coding (VVC).

[0006] CWG-B023 recommends using a reference motion vector (MV) candidate library in the AVM reference software. The reference MV candidate library can be used as a buffer for collecting reference MV candidates. According to the aforementioned standard, the reference MV candidate library is updated using the MVs used by the coded blocks. However, the reference MV candidate library update process does not consider MV candidates from the same superblock, leading to suboptimal results because the reference process fails to consider other possible orientations and blocks.

[0007] Therefore, a method, system, device, and / or non-volatile storage medium is needed to improve the generation and updating of reference MV candidate libraries in order to improve video coding. Summary of the Invention

[0008] According to an embodiment, a method for decoding a video stream can be provided. The method, which can be executed by at least one processor, may include: retrieving from a reference motion vector candidate library a plurality of motion vector prediction values ​​associated with a current block, the retrieved plurality of motion vector prediction values ​​being associated with a plurality of decoded blocks, and the plurality of decoded blocks belonging to the same superblock as the current block; determining motion vectors associated with the current block based on the retrieved plurality of motion vector prediction values; updating the reference motion vector candidate library by inserting the motion vectors associated with the current block into the reference motion vector candidate library; and decoding the current block based on the determined motion vectors associated with the current block.

[0009] According to an embodiment, an apparatus for decoding a video stream can be provided. The apparatus may include: at least one memory configured to store program code; and at least one processor configured to read the program code and execute the method for decoding the video stream as instructed by the program code.

[0010] According to an embodiment, a method for encoding a video stream can be provided for generating a video stream, comprising: retrieving from a reference motion vector candidate library multiple motion vector prediction values ​​associated with a current block, wherein the retrieved multiple motion vector prediction values ​​are associated with multiple encoded blocks and the multiple encoded blocks belong to the same superblock as the current block; determining motion vectors associated with the current block based on the retrieved multiple motion vector prediction values; updating the reference motion vector candidate library by inserting the motion vectors associated with the current block into the reference motion vector candidate library; and encoding the current block based on the determined motion vectors associated with the current block.

[0011] According to an embodiment, an apparatus for encoding a video stream can be provided for generating a video stream, the apparatus comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and perform the method for encoding the video stream as instructed by the program code.

[0012] According to an embodiment, a non-volatile computer-readable medium storing instructions may be provided. The instructions include one or more instructions that, when executed by one or more processors of a device, implement the method described above for decoding a video stream or the method described above for encoding a video stream.

[0013] According to an embodiment, a computer program product may be provided, including a computer program; when the computer program is executed by a processor, it implements the above-described method for decoding a video stream or the above-described method for encoding a video stream.

[0014] According to an embodiment, a method for storing a video stream can be provided, characterized in that the video stream is stored on a non-volatile computer-readable storage medium, wherein the video stream is generated according to the above-described method for encoding the video stream, or decoded based on the above-described method for decoding the video stream. Attached Figure Description

[0015] Figure 1AAn example of a segmentation tree under an AV1 and VPN framework according to an embodiment of the present disclosure is illustrated.

[0016] Figure 1B The illustration shows an example of block partitioning and tree structure using quadtree plus binary tree block partitioning according to an embodiment of the present disclosure.

[0017] Figure 1C Examples of vertical center-side ternary tree segmentation and horizontal center-side ternary tree segmentation according to embodiments of the present disclosure are illustrated.

[0018] Figure 1D An example of a search point for a merging pattern with motion vector difference is illustrated according to an embodiment of the present disclosure.

[0019] Figure 1E An example of a spatial motion vector neighbor according to an embodiment of the present disclosure is illustrated.

[0020] Figure 1F An example of motion field estimation by linear projection according to an embodiment of the present disclosure is illustrated.

[0021] Figure 1G An example of block location for deriving time motion vector prediction values ​​according to an embodiment of the present disclosure is illustrated.

[0022] Figure 1H An example of additional motion vector candidate generation for a block with a single reference, according to an embodiment of the present disclosure, is illustrated.

[0023] Figure 1I An example of additional motion vector candidate generation for a block with a composite reference according to an embodiment of the present disclosure is illustrated.

[0024] Figure 2 The illustration shows a prior art reference motion vector candidate update process according to an embodiment of the present disclosure.

[0025] Figure 3 The illustration shows a flowchart for constructing a candidate list of motion vectors according to an embodiment of the present disclosure.

[0026] Figure 4 This is a simplified block diagram of a communication system according to an embodiment of the present disclosure.

[0027] Figure 5 It is a layout diagram of video encoders and decoders in a streaming environment.

[0028] Figure 6 This is a functional block diagram of a video decoder according to an embodiment of the present disclosure.

[0029] Figure 7 This is a functional block diagram of a video encoder according to an embodiment of the present disclosure.

[0030] Figure 8 This is a flowchart of an example process for video encoding and decoding according to embodiments of the present disclosure.

[0031] Figure 9 This is a diagram of a computer system according to an embodiment of the present disclosure. Detailed Implementation

[0032] The features of the proposed methods, apparatus, and processes can be used individually or in combination. Embodiments of this disclosure relate to improvements in generating, maintaining, and updating a candidate library of reference motion vectors (MVs).

[0033] According to embodiments of this disclosure, candidate motion vector predictions (MVPs) can be updated at the coding block level, rather than at the superblock candidate MVP level, or in addition to updates at the superblock candidate MVP level, they can also be updated at the coding block level, where the candidate MVP can be updated at the superblock level. The advantage of coding block-level candidate MVPs is that more candidate MVPs from the same superblock can be used to more effectively predict the MV of the current block compared to using farther candidate MVPs from superblocks to the left, above, or even further away. Furthermore, in addition to updating candidate MVPs at the coding block level, candidate MVPs can also be updated at a specific level.

[0034] According to one embodiment, the MV of a coded block can be updated directly in the reference MV candidate library after parsing the MV of the coded block. The MV of the coded block can then be used as an MVP candidate for any subsequent coded block. In the prior art, the MV of a block is updated or inserted into the reference MV candidate library only after the entire superblock has been parsed, which is an inefficient use of the reference MV candidate library.

[0035] According to one embodiment of this disclosure, the reference MV candidate library can be updated after blocks in a predefined region are parsed or at a certain level. For example, the reference MV candidate library can be updated when all blocks in a 64×64 non-overlapping region are parsed or one level after the superblock level of a quadtree (QT). During encoding or decoding, after all coded blocks within this region are parsed, all MVs within the predefined region or QT can be updated to the reference MV candidate library and made available for reference.

[0036] The advantages of the above embodiments are that the reference MV library does not need to be updated very frequently. In some hardware designs, the reference MV library is not stored in a cache, which reduces latency. Additionally, the above embodiments avoid updating MVs that are too close to the current coding block. These MVs may overlap with the Spatial Motion Vector Prediction (SMVP) of the current block.

[0037] The above embodiments can be combined. As an example, two or more reference MV libraries are maintained. A first reference MV candidate library can be updated at the coding block level, and a second reference MV candidate library can be updated at a predefined level. During reference, the use of the first or second library as a reference can be implicitly selected or explicitly signaled. As a non-limiting example, two reference MV libraries are maintained. One library can be updated at the coding block level, i.e., immediately after a block MV is parsed, the MV of the parsed block is inserted into that library (e.g., the first library). Another library can be updated at the superblock level, i.e., after all coding blocks of the current block's superblock are parsed, the MVs of those coding blocks can be inserted into the other library. During reference, in one example, if the block is larger than a certain block size (e.g., 16×16), the library updated at the superblock level can be used; otherwise, if the block size is less than or equal to a certain block size (e.g., 16×16), the library updated at the block level can be used. It should be understood that combinations of multiple libraries or libraries of several different specific levels (e.g., 32×32 and 64×64) are also possible.

[0038] Embodiments of this disclosure also relate to a reference order for restoring a library of MV candidates. In related art, the reference to a library of MV candidates is always from the end of the library to the beginning. According to embodiments of this disclosure, the reference order for a library of MV candidates can be designed to be from the beginning to the end of the library. The advantage of this restoration order is that MV candidates further back in the list may be more relevant, which may be beneficial for certain content types, such as screen content or content that does not have translational motion. It should be understood that combinations of restoration references and references from the end of the library to the beginning are also possible. As an example, flags or conditions (e.g., block size, QP value, hash hit rate of screen content) can be used to determine which reference order can be used.

[0039] According to one embodiment, the levels, flags, conditions, or indexes of the reference MV candidate library can be signaled using high-level syntax, which includes, but is not limited to, sequence parameter sets, video parameter sets, image parameter sets, strip headers, frame headers, APS, and tile headers.

[0040] In one embodiment, the MV of a coded block can be conditionally updated to the reference MV candidate library directly after parsing the MV of that coded block, making the MV usable as an MVP candidate for any subsequent coded block. Other conditions can also be used individually or in combination, including but not limited to whether the parsed MV is associated with a block encoded in a specific encoding mode, or whether the parsed MV is associated with a block larger than, less than, or equal to a predefined block size. Compared to related techniques that require updating the block's MV to the reference MV candidate library after parsing the entire superblock, the embodiments described herein utilize the reference MV candidate library more efficiently because not every parsed MV can be used to update the reference MV candidate library; some MVs can be parsed but not used to update the reference MV candidate library, thereby reducing latency and improving the efficiency of the reference MV candidate library.

[0041] Block splitting in VP9 and AV1

[0042] Figure 1A Figure 1100 shows an example of a split tree under VP9 and AV1. Figure 1A As shown in the upper part, VP9 can use 4-way split trees starting from the 64×64 level down to the 4×4 level, with some additional restrictions on 8×8 blocks. It can be understood that a split designated as "R" refers to recursion—the same split tree can be repeated at a lower rate until the lowest 4×4 level is reached.

[0043] AV1 can not only expand the segmentation tree to, for example Figure 1A The lower half of the diagram shows a 10-way architecture, and AV1 also increases the maximum size (referred to as the superblock in the VP9 / AV1 syntax) starting from 128×128. It can be understood that this 10-way architecture can include 4:1 / 1:4 rectangular partitions that do not exist in VP9. These rectangular partitions do not need to be further subdivided. Additionally, because 2×2 chroma inter-frame prediction is now possible in some cases, AV1 adds more flexibility for using partitions below 8×8 levels.

[0044] Block splitting in HEVC

[0045] In HEVC, coding tree units (CTUs) are partitioned into coding units (CUs) using a quadtree (QT) structure represented as a coding tree to accommodate various local characteristics. The decision of whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode a picture region is made at the CU level. Each CU can be further divided into one, two, or four prediction units (PUs) depending on the PU partitioning type. Within a PU, the same prediction process is applied, and relevant information is sent to the decoder based on the PU. After obtaining the residual block by applying the prediction process based on the PU partitioning type, the CU can be partitioned into transform units (TUs) according to another quadtree structure (such as the coding tree of the CU). A characteristic of the HEVC structure is that it has multiple partitioning concepts including CUs, PUs, and TUs. In HEVC, the shape of a CU or TU can only be square, while for inter-frame prediction blocks, the shape of a PU can be square or rectangular. In HEVC, a coding block can be further divided into four square sub-blocks, and a transform is performed on each sub-block, i.e., a TU. Each TU can be further recursively partitioned (using quadtree partitioning) into smaller TUs, a process known as residual quadtree partitioning (RQT). At image boundaries, HEVC employs implicit quadtree partitioning, ensuring that blocks maintain quadtree partitions until their size fits the image boundaries. A key feature of the HEVC structure is its multiple partitioning concepts, including CU, PU, ​​and TU.

[0046] Block segmentation in Universal Video Coding (VVC)

[0047] A block partitioning structure using quadtrees (QT) and binary trees (BT)

[0048] The QTBT structure can incorporate multiple segmentation types, meaning it can remove the separation between CU, PU, ​​and TU concepts and support greater flexibility in CU segmentation shapes. In a QTBT block structure, the CU can have a square or rectangular shape. For example... Figure 1BAs shown, using tree 1205, the coding tree unit (CTU) is first segmented through a quadtree structure. The leaf nodes of the quadtree are further segmented through a binary tree structure. There are two types of segmentation in the binary tree segmentation: symmetrical horizontal segmentation and symmetrical vertical segmentation. The leaf nodes of the binary tree are called coding units (CUs), and this segmentation is used for prediction and transform processing without any further segmentation. Therefore, CUs, PUs, and TUs have the same block size in the QTBT coding block structure. In JEM, a CU can consist of coding blocks (CBs) of different color components; for example, in the case of P-stripes and B-stripes in a 4:2:0 chroma format, a CU contains one luma CB and two chroma CBs. Sometimes, a CU can also consist of CBs of a single component; for example, in the case of I-stripes, a CU contains only one luma CB or only two chroma CBs. The following parameters can be defined for the QTBT partitioning scheme: CTU size: the size of the root node of the quadtree; the same concept as in HEVC; MinQTSize: the minimum allowed size of the leaf nodes of the quadtree; MaxBTSize: the maximum allowed size of the root node of the binary tree; MaxBTDepth: the maximum allowed depth of the binary tree; and MinBTSize: the minimum allowed size of the leaf nodes of the binary tree.

[0049] In an example of a QTBT segmentation structure, the CTU size can be set to 128×128 luminance samples, each with two corresponding 64×64 chrominance sample blocks. The MinQTSize can be set to 16×16, the MaxBTSize to 64×64, and the MinBTSize (for both width and height) can be set to 4×4, with MaxBTDepth set to 4. Quadtree segmentation can be applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes ranging from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If a quadtree leaf node is 128×128, it will not be further segmented by the binary tree because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, the quadtree leaf node can be further segmented by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree, and the binary tree depth is 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), further segmentation is not considered. When a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal segmentation is considered. Similarly, when a binary tree node has a height equal to MinBTSize, no further vertical segmentation is considered. Leaf nodes of the binary tree are further processed through prediction and transformation without any further segmentation. In JEM, the maximum CTU size is 256×256 luminance samples.

[0050] like Figure 1B As shown, block 1205 illustrates an example of block segmentation using QTBT, and Figure 1B The binary tree 1210 illustrates the corresponding tree representation. Solid lines indicate quadtree partitions, and dashed lines indicate binary tree partitions. In each partition (i.e., non-leaf) node of the binary tree, a flag can be signaled to indicate which partition type (i.e., horizontal or vertical) is used. Here, 0 indicates a horizontal partition, and 1 indicates a vertical partition. For quadtree partitions, it is not necessary to indicate the partition type because quadtree partitions always divide blocks horizontally and vertically to produce four sub-blocks of equal size.

[0051] Furthermore, the QTBT scheme supports the flexibility of having separate QTBT structures for luma and chroma. Currently, in existing technologies, for P-strip and B-strip, the luma CTB and chroma CTB within a single CTU share the same QTBT structure. However, for I-strip, the luma CTB is divided into CUs using a QTBT structure, and the chroma CTB is divided into chroma CUs using a separate QTBT structure. This means that a CU in an I-strip can consist of a block of code for the luma component or two blocks of code for the chroma components, while a CU in a P-strip or B-strip consists of blocks of code for all three color components.

[0052] In HEVC, inter-frame prediction for small blocks can be restricted to reduce memory accesses for motion compensation, allowing bidirectional prediction to be unsupported for 4×8 and 8×4 blocks, and inter-frame prediction is not used for 4×4 blocks. These restrictions are removed in QTBT implemented in JEM-7.0.

[0053] Using a block partitioning structure of a ternary tree (TT)

[0054] In VVC, a Multi-Type Tree (MTT) structure can be included, which adds a horizontal center-side ternary tree and a vertical center-side ternary tree on top of the QTBT, such as... Figure 1C The figures are shown at blocks 1305 and 1310, respectively. The advantage of TT partitioning is that it complements quadtree and binary tree partitioning; TT partitioning can also capture objects located at the block center, while quadtrees and binary trees are always partitioned along the block center. Furthermore, since the width and height of the proposed TT partitioning are always powers of 2, no additional transformations are required. The design of the second-order tree is primarily driven by the need for reduced complexity. Theoretically, the complexity of traversing a tree is T. D Where T represents the number of partition types and D is the depth of the tree.

[0055] Merging Mode with Motion Vector Difference (MMVD)

[0056] In addition to the merge mode (where implicitly derived motion information is directly used to generate prediction samples for the current CU), a merge mode with motion vector difference (MMVD) is also introduced in VVC. The MMVD flag can be signaled immediately after sending the skip flag and merge flag to indicate whether the MMVD mode is used for the CU.

[0057] In MMVD, after selecting a merge candidate, the merge candidate can be further refined using signaled MVD information. This further information can include a merge candidate flag, an index specifying the motion amplitude, and an index indicating the motion direction. In MMVD mode, one of the top two candidates in the merge list is selected as the MV basis. The merge candidate flag can be signaled to specify which one to use.

[0058] The distance index specifies the motion amplitude information and indicates a predefined offset from the starting point. For example... Figure 1D As shown, an offset can be added to the horizontal or vertical component of the starting MV. The relationship between the distance index and the predefined offset can be specified in Table 1.

[0059] Table 1 – Relationship between Distance Index and Predefined Offset

[0060]

[0061] The direction index indicates the direction of MVD relative to the starting point. The direction index can represent the four directions shown in Table 2.

[0062] Table 2 – Sign of MV Offset Specified by Direction Index

[0063] Directional IDX 00 01 10 11 x-axis + - not applicable not applicable y-axis not applicable not applicable + -

[0064] It should be noted that the meaning of the MVD sign can vary depending on the information of the starting MV. When the starting MV is a single-prediction MV or a double-prediction MV, where both lists point to the same side of the current image, the sign in Table 2 specifies the sign of the MV offset added to the starting MV. For example, this applies when both reference POCs are greater than or less than the current image's POC. When the starting MV is a bidirectional prediction MV and the two MVs point to different sides of the current image, and the POC difference in list 0 is greater than the POC difference in list 1, the sign in Table 2 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of list 1 MV has the opposite value. For example, if one reference POC is greater than the current image's POC, and the other reference POC is less than the current image's POC, then the sign of the MV offset added to the list 0 MV component of the starting MV and the sign of list 1 MV have the opposite value. Otherwise, if the POC difference in list 1 is greater than that in list 0, then the sign in Table 2 specifies the sign of the MV offset added to the list 1 MV component of the starting MV, and the sign of list 0 MV has the opposite value.

[0065] MVD can be scaled based on the difference in POC in each direction. If the difference in POC is the same in both lists, no scaling is performed. Otherwise, if the difference in POC in list 0 is greater than the difference in POC in list 1, the MVD of list 1 can be scaled. If the difference in POC in L1 is greater than that in L0, the MVD of list 0 can be scaled in the same way. If the initial MV is a single prediction, the MVD can be added to the available MVs.

[0066] Symmetric MVD coding

[0067] In VVC, in addition to normal one-way and two-way prediction MVD signaling, a symmetric MVD mode for two-way MVD signaling can also be applied. In the symmetric MVD mode, the reference image indices of both List 0 and List 1, as well as the motion information of MVD in List 1, can be derived without being signaled.

[0068] The decoding process for a strip-level symmetric MVD mode can be as follows. At the strip level, derive the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1. If mvd_l1_zero_flag is 1, then BiDirPredFlag can be set to 0. Otherwise, if the most recent reference image in list 0 and the most recent reference image in list 1 form a forward and backward reference image pair or a backward and forward reference image pair, then BiDirPredFlag can be set to 1, and both list 0 and list 1 reference images can be short-term reference images. Otherwise, BiDirPredFlag is set to 0.

[0069] The decoding process for symmetric MVD mode at the CU level can be as follows. At the CU level, if the CU is double predictive coded and BiDirPredFlag equals 1, the symmetric mode flag indicates whether symmetric mode can be used or whether explicit signaling is not allowed. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 are explicitly signaled. The reference indices of list 0 and list 1 are set to equal the reference image pair, respectively. MVD1 is set to equal to (-MVD0).

[0070] Inter-frame mode coding in CWG-B018

[0071] In AV1, for each coded block in an inter-frame frame, if the current block's mode is not skip mode but inter-frame coding mode, another flag can be signaled to indicate whether a single reference mode or a composite reference mode is used for the current block.

[0072] A single mode can include a prediction block generated from a motion vector in a single reference mode. The following modes can be signaled in a single reference case: (1) NEARMV – using one of the motion vector prediction values ​​(MVPs) in the list indicated by the Dynamic Reference List (DRL) index; (2) NEWMV – using one of the motion vector prediction values ​​(MVPs) in the list signaled by the DRL index as a reference and applying an increment to the MVP; and (3) GLOBALMV – using a motion vector based on frame-level global motion parameters.

[0073] The prediction block generated by weighted averaging of two prediction blocks can be derived from two motion vectors in the composite reference mode. The following modes can be signaled in the case of composite reference: (1) NEAR_NEARMV - using one of the motion vector prediction values ​​(MVPs) in the list signaled by the DRL index; (2) NEAR_NEWMV - using one of the motion vector prediction values ​​(MVPs) in the list signaled by the DRL index as a reference and sending an incremental MV for the second MV; (3) NEW_NEARMV - using one of the motion vector prediction values ​​(MVPs) in the list signaled by the DRL index as a reference and sending an incremental MV for the first MV; (4) NEW_NEWMV - using one of the motion vector prediction values ​​(MVPs) in the list signaled by the DRL index as a reference and sending an incremental MV for both MVs; and (5) GLOBAL_GLOBALMV - using the MV from each reference based on their frame-level global motion parameters.

[0074] Motion Vector Differential Coding in AV1

[0075] AV1 allows 1 / 8 pixel motion vector precision (or accuracy), and the following syntax can be used to signal motion vector differences in reference frame list 0 or list 1: (1) mv_joint specifies which components of the motion vector difference are non-zero—0 indicates that there is no non-zero MVD along the horizontal or vertical direction, 1 indicates that there is a non-zero MVD only along the horizontal direction, 2 indicates that there is a non-zero MVD only along the vertical direction, and 3 indicates that there is a non-zero MVD along both the horizontal and vertical directions; (2) mv_sign specifies whether the motion vector difference is positive or negative; (3) mv_class specifies the class of the motion vector difference. As shown in Table 3, a higher class means a larger magnitude of the motion vector difference; (4) mv_bit specifies the integer part of the offset between the motion vector difference and the starting magnitude for each MV class; (5) mv_fr specifies the first two fractions of the motion vector difference; and (6) mv_hp specifies the third fraction of the motion vector difference.

[0076] Table 3: Amplitude Categories of Motion Vector Difference

[0077]

[0078]

[0079] Adaptive MVD Resolution in CWG-B092

[0080] For NEW_NEARMV and NEAR_NEWMV modes, the precision of MVD can depend on the associated class and magnitude of the MVD. First, fractional MVD is allowed only when the MVD magnitude is equal to or less than one pixel. Second, only one MVD value is allowed when the value of the associated MV class is equal to or greater than MV_CLASS_1. For MV classes 1 (MV_CLASS_1), 2 (MV_CLASS_2), 3 (MV_CLASS_3), 4 (MV_CLASS_4), or 5 (MV_CLASS_5), the MVD values ​​in each MV class are derived as 4, 8, 16, 32, and 64, respectively. Additionally, if the current block is encoded in NEW_NEARMV or NEAR_NEWMV mode, one context can be used to signal either mv_joint or mv_class. Otherwise, another context can be used to signal either mv_joint or mv_class.

[0081] Table 4: Adaptive MVD in each MV amplitude category

[0082] 25. Music Video Category 26. MVD Amplitude 27.MV_CLASS_0 28.(0，1]，{2} 29.MV_CLASS_1 30.{4} 31.MV_CLASS_2 32.{8} 33.MV_CLASS_3 34.{16} 35.MV_CLASS_4 36.{32} 37.MV_CLASS_5 38.{64} 39.MV_CLASS_6 40.{128} 41.MV_CLASS_7 42.{256} 43.MV_CLASS_8 44.{512} 45.MV_CLASS_9 46.{1024} 47.MV_CLASS_10 48.{2048}

[0083] Joint MVD Coding in CWG-B092 (JMVD)

[0084] A new inter-frame coding mode named JOINT_NEWMV can be applied to indicate whether the MVDs of both reference lists are jointly signaled. If the inter-frame prediction mode is equal to the JOINT_NEWMV mode, the MVDs of reference list 0 and reference list 1 are jointly signaled. Therefore, only one MVD (named joint_mvd) is signaled and sent to the decoder, and the incremental MVs for reference list 0 and reference list 1 are derived from joint_mvd.

[0085] The JOINT_NEWMV mode signals together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. No additional context is added.

[0086] When the JOINT_NEWMV mode is signaled and the POC distances between the two reference frames and the current frame are different, the MVD is scaled for either reference list 0 or reference list 1 based on the POC distance. Specifically, the distance between reference list 0 and the current frame can be td0, and the distance between reference list 1 and the current frame is marked as td1. If td0 is equal to or greater than td1, then joint_mvd is directly used for reference list 0, and the MVD for reference list 1 is derived from joint_mvd based on equation (1).

[0087]

[0088] Otherwise, if td1 is equal to or greater than td0, then joint_mvd is used directly for reference list 1, and mvd for reference list 0 is derived from joint_mvd based on equation (2).

[0089]

[0090] Improvements to Adaptive MVD Resolution in CWG-C011

[0091] A new inter-frame coding mode named AMVDMV can be added to a single reference case. When the AMVDMV mode is selected, it indicates that AMVD is applied to signal MVD. In JOINT_NEWMV mode, a flag named amvd_flag is added to indicate whether AMVD is applied to the joint MVD coding mode. When adaptive MVD resolution is applied to the joint MVD coding mode named joint AMVD coding, the MVD of two reference frames can be jointly signaled, and the precision of the MVD is implicitly determined by the MVD amplitude. Otherwise, the MVD for two (or more) reference frames is jointly signaled, and regular MVD coding is applied.

[0092] Adaptive Motion Vector Resolution (AMVR) in CWG-C012 and CWG-C020

[0093] AMVR was first proposed in CWG-C012, which supports a total of 7 MV accuracies (8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8). For each prediction block, the AVM encoder can search all supported accuracies and signal the best accuracies to the decoder.

[0094] To reduce encoder runtime, two precision sets are supported. Each precision set contains four predefined precisions. The precision set can be adaptively selected at the frame level based on the maximum precision value of the frame. Similar to AV1, the maximum precision can be signaled in the frame header. Table 5 summarizes the supported precision values ​​based on the frame-level maximum precision.

[0095] Table 5: MV accuracy of two centralized supports

[0096] Frame-level maximum precision Supported MV precision 1 / 8 1 / 8、1 / 2、1、4 1 / 4 1 / 4、1、4、8

[0097] In current AVM software (similar to AV1), there are frame-level flags indicating whether the frame's MV includes subpixel precision. AMVR is enabled only when the value of the `cur_frame_force_integer_mv` flag is 0. In AMVR, if the block's precision is less than the maximum precision, the motion model and interpolation filter are not signaled. If the block's precision is less than the maximum precision, the motion model is inferred as translational motion, and the interpolation filter can be inferred as a regular interpolation filter. Similarly, if the block's precision is 4 pixels or 8 pixels, the inter-frame / intra-frame mode is not signaled and is inferred as 0.

[0098] List of motion vector predictions in AV1 and AVM

[0099] The AVM design adds additional Spatial Motion Vector Predictions (SMVPs) (neighboring SMVPs and non-neighboring SMVPs), Temporal Motion Vector Predictions, additional MV candidates from AV1, additionally derived MVPs, and reference library MVPs. Fixed-size stacks, referred to as the Motion Vector Prediction List, are generated at both the encoder and decoder ends to store the MVPs.

[0100] Spatial Motion Vector Prediction (SMVP)

[0101] Spatial motion vector (MV) predictions are derived from spatial neighbor blocks, which include adjacent spatial neighbor blocks (the direct neighbors of the current block above and to the left) and non-adjacent spatial neighbor blocks (those that are close to but not directly adjacent to the current block). Figure 1E The diagram illustrates an example of a set of spatial neighbor blocks for a luma block, where each spatial neighbor block is an 8×8 block. Spatial neighbor blocks are examined to find one or more MVs associated with the same reference frame index as the current block. For the current block, the search order of the spatial neighbor 8×8 luma blocks is as follows: Figure 5 As shown in numbers 1-8: (1) Check the top adjacent rows from left to right; (2) Check the left adjacent columns from top to bottom; (3) Check the top right neighboring block; (4) Check the top left neighboring block; (5) Check the first top non-adjacent rows from left to right; (6) Check the first left non-adjacent columns from top to bottom; (7) Check the second top non-adjacent rows from left to right; and (8) Check the second left non-adjacent columns from top to bottom.

[0102] Before TMVP, adjacent candidates ( Figure 1E Add items 1-3 from the list to the MV prediction list. After TMVP, add non-neighboring candidates (also known as external candidates, i.e.) Figure 1E Candidates 4-8 in the list are added to the MV prediction list. All SMVP candidates should have the same reference image as the current block. That is, if the current block has a single reference image, and an MVP candidate has a single reference image that is the same as the current block's reference image, or if an MVP candidate has a composite reference image (two reference images) and one of these two reference images is the same as the current block's reference image, then that MVP candidate is added to the MV prediction list. If the current block has two reference images, then that MVP candidate is added to the prediction list only if it has two reference images and both of these reference images are the same as the current block's reference image.

[0103] Time Motion Vector Prediction (TMVP)

[0104] In addition to spatial neighbor blocks, co-located blocks in reference frames can be used to derive MV predictions, known as temporal MV predictions. To generate temporal MV predictions, the MV of the reference frame is first stored along with the reference index associated with each reference frame. Then, for each 8×8 block of the current frame, the MV of the reference frames whose trajectories cross the 8×8 block is identified and stored in the temporal MV buffer along with the reference frame index. For inter-frame prediction using a single reference frame, regardless of whether the reference frame is forward or backward, the MV is stored in 8×8 units for performing temporal motion vector predictions for future frames. For composite inter-frame prediction, only the forward MV is stored in 8×8 units for performing temporal motion vector predictions for future frames.

[0105] refer to Figure 1G The MV (i.e., MVref 1650) of reference frame 1(R1) 1620 is pointed to from frame 1(R1) 1620. In doing so, MVref 1650 passes through the 8×8 block of the current frame (the block in current frame 1615 and the block in reference frame 0 1610). The MVref is stored in the temporal MV buffer associated with that 8×8 block. During the motion projection process used to derive the temporal MV prediction values, reference frames can be scanned in a predefined order: LAST_FRAME, BWDREF_FRAME, ALTREF_FRAME, ALTREF2_FRAME, and LAST2_FRAME. MVs from higher-indexed reference frames (in scan order) do not replace previously identified MVs assigned by lower-indexed reference frames (in scan order).

[0106] Given predefined block coordinates, the associated MV stored in the temporal MV buffer is identified and projected onto the current block to derive the temporal MV prediction value pointing from the current block to its reference frame, for example... Figure 1F MV0 in the middle.

[0107] refer to Figure 1G This shows the predefined block locations used to derive the temporal MV predictions for a 16×16 block. Valid temporal MV predictions are checked for up to seven blocks. Temporal MV predictions are checked after adjacent spatial MV predictions but before non-adjacent spatial MV predictions.

[0108] For the derivation of MV predictions, all spatial and temporal MV candidates can be aggregated, and each prediction can be assigned a weight determined during the scanning of spatial and temporal neighbor blocks. Based on the associated weights, the candidates can be classified and sorted, and up to four candidates can be identified and added to the MV prediction list. This MV prediction list is also known as the Dynamic Reference List (DRL), which is further used in dynamic MV prediction patterns, as described in the next section.

[0109] Additional search for MVP candidates

[0110] If the MVP list is still not full, an additional search will be performed, and additional MVP candidates will be used to populate the MVP list. Additional MVP candidates include, for example, global MV, zero MV, and composite MVs without scaling.

[0111] MVP Candidate Reordering Process

[0112] Neighboring SMVP candidates, TMVP candidates, and non-neighboring SMVP candidates added to the MVP list will be reordered. Based on the current design in AV1 and AVM, the reordering process is based on the weight of each candidate. The weights of candidates are predefined depending on the overlap between the current block and the candidate blocks.

[0113] Exported MVP candidates

[0114] CWG-B049 proposes that the exported MVP candidate be used in the AVM reference software, which includes exported MVPs for single reference images and composite modes.

[0115] Single frame prediction

[0116] If the reference frame of a neighboring block is different from the reference frame of the current block, but they are in the same direction, a time scaling algorithm can be used to scale its motion vector (MV) to that reference frame in order to form the MVP of the current block's motion vector. For example... Figure 1H As shown, mv1 1855 from the neighboring block (shaded block) can be used to derive the MVP of the current block's motion vector mv0 1850 at time scaling.

[0117] Composite inter-frame prediction

[0118] The MVP of the current block is derived by synthesizing MVs from different neighboring blocks, but the reference frame for the synthesized MV must be the same as that of the current block. For example... Figure 1I As shown, the composite MV(mv2 1960, mv3 1965) has the same reference frame as the current block, but they come from different neighboring blocks.

[0119] Reference motion vector candidate library

[0120] Each buffer corresponds to a unique reference frame type, which may be a single or a pair of reference frames, covering single inter-frame mode and composite inter-frame mode, respectively. All buffers are the same size. When a new MV is added to a full buffer, an existing MV can be evicted to make room for the new MV.

[0121] In addition to the reference MV candidates generated using the regular AV1 reference MV list, the coded block can also reference the MV candidate library used to collect reference MV candidates. After the superblock is encoded, the MV library can be updated using the MVs used by the coded blocks of the superblock.

[0122] Each tile can have an independent reference MV library that can be used by all superblocks within that tile. At the beginning of encoding each tile, the corresponding library can be cleared. Subsequently, when encoding each superblock within that tile, MVs from the library can be used as MV reference candidates. At the end of encoding a superblock, the library can be updated.

[0123] Library Update

[0124] like Figure 2 As shown in Figure 200, the library update process can be based on superblocks. That is, after encoding the superblock, the first (up to 64) candidate MVs used by each encoded block within the superblock are added to the library. A pruning process can also be involved during the update.

[0125] Library Reference

[0126] After completing a regular AV1 or new AV2 reference MV candidate scan, if there are free slots in the candidate list, the codec can refer to the MV candidate library (in buffers with matching reference frame types) to obtain additional MV candidates. Starting from the end of the buffer and working backwards to the beginning, if an MV is not found in the list, the MV from the library buffer can be appended to the candidate list.

[0127] MVP List Building Process in State-of-the-Art Design

[0128] In existing technologies, such as Figure 3 As shown in flowchart 300, the MVP list can be constructed using full pruning. Examples may include: operation 305, which includes adding neighboring SMVPs; operation 310, which includes adding TMVPs; operation 315, which includes adding non-neighboring SMVPs; operation 320, which includes adding a reordering process for existing candidates; operation 325, which includes adding exported candidates; operation 330, which includes adding additional MVP candidates; and finally, operation 355, which includes adding candidates from a reference MVP candidate library.

[0129] Figure 4An exemplary block diagram of a communication system 400 according to an embodiment of the present disclosure is shown. The communication system 400 may include at least two terminals 410 and 420, which can communicate with each other via a network 450. For one-way data transmission, a first terminal 410 may encode video data locally for transmission to another terminal 420 via the network 450. A second terminal (220) may receive encoded video data from the other terminal via the network 450, decode the encoded video data, and display the recovered video data. One-way data transmission is common in applications such as media services.

[0130] Figure 4 A second pair of terminals 430 and 440 supporting bidirectional transmission of encoded video data is shown, which may occur, for example, during a video conference. For bidirectional data transmission, in one example, each terminal 430, 440 may encode video data acquired at a local location for transmission to another terminal device via network 450. Each terminal 430, 440 may also receive encoded video data transmitted by the other terminal, decode the encoded video data, and display the recovered video data on a local display device.

[0131] exist Figure 4 In the embodiments described, terminals 410, 420, 430, and 440 may be servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 450 refers to any number of networks, including, for example, wired and / or wireless communication networks, that transmit encoded video data between terminals 410, 420, 430, and 440. Communication network 450 may exchange data in circuit-switched and / or packet-switched channels. This network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this disclosure, unless explained below, the architecture and topology of network 450 may be irrelevant to the operation of this disclosure.

[0132] As an example of the disclosed subject matter, Figure 5 The placement of a video encoder and video decoder in a streaming environment, such as streaming system 500, is illustrated. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0133] The streaming system may include an acquisition subsystem 513, which may include a video source 501 such as a digital camera, which creates, for example, an uncompressed video sample stream 502. The video sample stream 502 is depicted as a thick line to emphasize a high data volume compared to an encoded video stream. The video sample stream 502 may be processed by an encoder 503 coupled to the camera 501. The encoder 503 may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video stream 504 is depicted as a thin line to emphasize a lower data volume compared to the sample stream, and may be stored on a streaming server 505 for future use. One or more streaming clients 506, 508 may access the streaming server 505 to retrieve copies 507, 509 of the encoded video stream 504. Client 506 may include a video decoder 510. Video decoder 510 decodes an incoming copy of the encoded video stream and produces an output video sample stream 511 that can be displayed on display 512 or other presentation device (not shown). In some streaming systems, the encoded video stream 504, 507, 509 may be encoded according to certain video coding / compression standards. Examples of these standards include ITU-T H.265. In embodiments, the video codec standard under development is informally referred to as Versatile Video Coding (VVC), and the disclosed subject matter can be used within the context of the VVC standard.

[0134] Figure 6 This can be a functional block diagram of the video decoder 510 according to an embodiment of the present invention.

[0135] Receiver 610 may receive one or more encoded video sequences to be decoded by decoder 510; in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the others. Encoded video sequences may be received from channel 612, which may be a hardware / software link to a storage device storing the encoded video data. Receiver 610 may receive encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective usage entities (not shown). Receiver 610 may separate the encoded video sequences from other data. To prevent network jitter, buffer memory 615 may be coupled between receiver 610 and entropy decoder / parser 620 (hereinafter referred to as "parser 620"). Buffer memory 615 may not be required when receiver 610 receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, or the buffer memory may be made smaller. Of course, in order to be used on business packet networks such as the Internet, a buffer memory 615 may also be required, which may be relatively large and have an adaptive size.

[0136] The video decoder 510 may include a parser 620 to reconstruct symbols (421) from an entropy-encoded video sequence. These symbols include information for managing the operation of the video decoder 510, and potential information for controlling a display device such as a display 512, which is not part of the decoder but may be coupled to it. Figure 6As shown in the diagram. The control information for the display device may be a parameter set fragment of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI) (not shown). Parser 620 can parse / decode the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 620 can extract a subgroup parameter set from the encoded video sequence for at least one subgroup of pixels in the subgroups used in the video decoder, based on at least one parameter corresponding to a group. Subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The entropy decoder / parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter QP values, motion vectors, and so on.

[0137] The parser 620 can perform entropy decoding / parsing operations on the video sequence received from the buffer memory 615 to create symbols 621. The parser 620 can receive encoded data and selectively decode specific symbols 621. Furthermore, the parser 620 can determine whether to provide specific symbols 621 to the motion compensation prediction unit 653, the scaler / inverse transform unit 651, the intra-frame prediction unit 652, or the loop filter 656.

[0138] Depending on the type of encoded video frames or portions thereof, such as inter-frame and intra-frame frames, inter-frame and intra-frame blocks, and other factors, the reconstruction of symbol 621 may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by the parser 620. For brevity, the flow of such subgroup control information between the parser 620 and the various units described below is not described.

[0139] In addition to the functional blocks already mentioned, decoder 510 can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide them into the functional units described below.

[0140] The first unit is the scaler / inverse transform unit 651. The scaler / inverse transform unit 651 receives quantization transform coefficients as symbols 621 from the parser 620, along with control information including the transform mode used, block size, quantization factor, and quantization scaling matrix. The scaler / inverse transform unit 651 can output blocks containing sample values, which can be input into the aggregator 655.

[0141] In some cases, the output samples of the scaler / inverse transform unit 651 may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit 652. In some cases, the intra-picture prediction unit 652 uses surrounding reconstructed information extracted from the currently reconstructed image 658 to generate blocks of the same size and shape as the block being reconstructed. In some cases, the aggregator 655 adds the predictive information generated by the intra-picture prediction unit 652 to the output sample information provided by the scaler / inverse transform unit 651 based on each sample.

[0142] In other cases, the output samples of the scaler / inverse transform unit 651 may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit 653 can access the reference image buffer 657 to extract samples for prediction. After motion compensation is performed on the extracted samples according to the symbols, these samples can be added to the output of the scaler / inverse transform unit 651 by the aggregator 655. In this case, these samples may be referred to as residual samples or residual signals, thereby generating output sample information. The motion compensation unit's acquisition of predicted samples from the address within the reference image buffer may be controlled by motion vectors, and the motion vectors are available to the motion compensation unit in the form of the symbols 621, which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​obtained from the reference image buffer when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0143] The output samples of aggregator 655 can be employed by various loop filtering techniques in loop filter unit 656. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream, and these parameters can be used as symbols 621 from parser 620 in loop filter unit 656. However, in other embodiments, video compression techniques may also respond to metadata obtained during decoding of a previously decoded portion of an encoded image or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0144] The output of the loop filter unit 656 can be a sample stream, which can be output to the display device 512 and stored in the reference image buffer 657 for subsequent inter-frame image prediction.

[0145] Once fully reconstructed, certain encoded images can be used as reference images for future predictions. For example, once an encoded image has been fully reconstructed and has been identified as a reference image by, for example, parser 620, the current reference image 658 can become part of the reference image buffer 657, and new current image memory can be reallocated before reconstructing subsequent encoded images begins.

[0146] The video decoder 510 can perform decoding operations according to predetermined video compression techniques, such as those in the ITU-T Rec.H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (e.g., the maximum reconstruction sampling rate measured in megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0147] In this embodiment, receiver 610 may receive additional redundant data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by video decoder 510 to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0148] Figure 7 A functional block diagram of a video encoder 503 according to an embodiment of the present disclosure is shown.

[0149] The video encoder 503 can receive video samples from a video source 501, which is not part of an encoder that can acquire video images to be encoded by the video encoder 503.

[0150] Video source 501 can provide a source video sequence in the form of a digital video sample stream encoded by encoder 503. This digital video sample stream can have any suitable bit depth, such as 8-bit, 10-bit, 12-bit, etc.; any color space, such as BT.601YCrCb, RGB, etc.; and any suitable sampling structure, such as YCrCb 4:2:0, YCrCb 4:4:4. In a media service system, video source 501 can be a storage device storing previously prepared video. In a video conferencing system, video source 501 can be a camera capturing local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed sequentially. A picture itself can be constructed as a spatial pixel array, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0151] According to an embodiment, encoder 503 can encode and compress images of a source video sequence into an encoded video sequence 743 in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of controller 750. Controller 750 controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller may include rate control related parameters: image skipping, quantizer, λ value of rate-distortion optimization techniques, image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can readily identify other functions of controller 750, as these functions may relate to encoder 503 optimized for a particular system design.

[0152] Some video encoders operate within an “encoding loop” readily recognizable to those skilled in the art. In a simplified description, the encoding loop may consist of the encoding portion of encoder 730, the “source encoder,” and a local decoder 733 embedded in encoder 503. The source encoder is then responsible for creating symbols based on the input image to be encoded. The local decoder 733 reconstructs the symbols to create sample data, which is also created by a remote decoder, because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video stream is lossless. The reconstructed sample stream is input to a reference image memory 734. Since the decoding of the symbol stream produces bit-precise results independent of whether the decoder is local or remote, the contents of the reference image buffer are also bit-precisely corresponding between the local and remote encoders. In other words, the reference image samples “seen” by the encoder’s prediction portion are exactly the same sample values ​​that the decoder will “see” during prediction. This fundamental principle of reference image synchronization, and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors, is well known to those skilled in the art.

[0153] The operation of the local decoder 733 can be combined with, for example, the above. Figure 6 The "remote" decoder described in detail is the same. However, another brief reference is available. Figure 7 When symbols are available and the entropy encoder 745 and the parser 620 are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the decoder 510, including the channel 612, receiver 610, buffer 615 and parser 620, may not be fully implemented in the local decoder 733.

[0154] It can then be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0155] As part of its operation, the source encoder 730 can perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence designated as "reference pictures." In this way, the encoding engine 732 encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which can be selected as a prediction reference for the input frame.

[0156] The local video decoder 733 can decode encoded video data that can be designated as a reference frame, based on symbols created by the source encoder 730. The encoding engine 732 can operate as a lossy process. When the encoded video data is available... Figure 7 When decoded at a video decoder not shown, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 7337 replicates the decoding process, which can be performed by the video decoder on the reference frame, and allows the reconstructed reference frame to be stored in the reference frame cache 734. In this way, the encoder 503 can locally store a copy of the reconstructed reference frame that shares content with the reconstructed reference frame to be obtained by the remote video decoder in the absence of transmission errors.

[0157] Predictor 735 can perform a prediction search against encoding engine 732. That is, for a new frame to be encoded, predictor 735 can search the reference image memory 734 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference frame motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new frame. Predictor 735 can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by predictor 735, it can be determined that the input frame may have prediction references obtained from multiple reference frames stored in the reference frame memory 734.

[0158] The controller 750 manages the encoding operations of the video encoder 730, including, for example, setting parameters and subgroup parameters for encoding video data.

[0159] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 745. The entropy encoder converts the symbols generated by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding.

[0160] Transmitter 740 can buffer the encoded video sequence created by entropy encoder 745, thereby preparing it for transmission via communication channel 760, which may be a hardware / software link to a storage device that will store the encoded video data. Transmitter 740 can combine the encoded video data from encoder 503 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0161] The controller 750 manages the operation of the encoder 503. During encoding, the controller 750 can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following frame types:

[0162] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are familiar with variations of I-pictures and their corresponding applications and characteristics.

[0163] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0164] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0165] Source images are typically spatially subdivided into multiple sample blocks, such as 4×4, 8×8, 4×8, or 16×16 sample blocks, and encoded block by block. These blocks can be predictively encoded with reference to other encoded blocks, determined based on the encoding assignment of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively encoded, or the blocks can be predictively encoded with reference to previously encoded blocks of the same image that are spatially or intra-frame predicted. Pixel blocks of a P-image can be non-predictively encoded with reference to a previously encoded reference image through spatial prediction or temporal prediction. Blocks of a B-image can be non-predictively encoded with reference to one or two previously encoded reference images through spatial prediction or temporal prediction.

[0166] Encoder 503 can perform encoding operations according to predetermined video coding techniques or standards, such as those specified in ITU-T H.265 Recommendation. In operation, encoder 503 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0167] In one embodiment, transmitter 740 may transmit additional data while transmitting encoded video. Video encoder 730 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplementary enhancement information (SEI) messages, fragments of visual usability information (VUI) parameter sets, etc.

[0168] Figure 8An exemplary process 800 for updating the reference motion vector candidate library by retrieving a motion vector predictor from a first reference motion vector candidate library is illustrated.

[0169] At operation 805, one or more motion vector predictions associated with the current block can be retrieved from a reference motion vector candidate library. The one or more motion vector predictions associated with the current block retrieved from the reference motion vector candidate library may include at least one or more motion vectors associated with one or more decoded blocks, and these decoded blocks may belong to the same superblock as the current block. In some embodiments, the one or more retrieved motion vector predictions associated with the current block are retrieved from the reference motion vector candidate library based on a recovered reference order. The indication of this reference order may be based on satisfied conditions. These conditions may be based on one of the following: the size of the current block; one or more quantization parameter values; or the hash hit rate of the screen content.

[0170] At operation 810, a motion vector associated with the current block can be determined based on one or more retrieved motion vector prediction values. At operation 815, the current block can be decoded based on the determined motion vector. In some embodiments, after decoding the current block based on the motion vector associated with the current block inserted into a reference motion vector candidate library, another block in the same superblock can be decoded. At operation 820, the reference motion vector candidate library can be updated by inserting the motion vector associated with the current block into the reference motion vector candidate library.

[0171] In some embodiments, at operation 825, updating the reference motion vector candidate library may include inserting the motion vector associated with the current block into the reference motion vector candidate library after decoding all blocks in the same superblock.

[0172] At operation 830, in some embodiments, updating the reference motion vector candidate library may include inserting the motion vectors associated with the current block into the reference motion vector candidate library after decoding all blocks at a predefined level of the quadtree associated with the current block. The predefined level of the quadtree may be the first level of the quadtree after the superblock level. The predefined level of the quadtree may be signaled using high-level syntax.

[0173] According to an embodiment, at operation 835, updating the reference motion vector candidate library may include inserting the motion vector associated with the current block into the reference motion vector candidate library after decoding all blocks in a predefined region surrounding the current block. The predefined region surrounding the current block may include a block-sized or fixed region such as 64×64, 32×64, etc.

[0174] In some embodiments, operations 805-835 may include the reference motion vector candidate library as a first reference motion vector candidate library, and another reference motion vector candidate library as a second reference motion vector candidate library. Updating the reference motion vectors may include updating the second reference motion vector candidate library by inserting the motion vectors associated with the current block into the second reference motion vector candidate library after decoding all blocks at a predefined level of the quadtree associated with the current block.

[0175] In some embodiments, based on the reference motion vector candidate library as a first reference motion vector candidate library and another reference motion vector candidate library as a second reference motion vector candidate library, when the size of the current block being decoded may be greater than a threshold, the current block can be decoded in operation 815 based on one or more first motion vectors from the first reference motion vector candidate library. In the same or another embodiment, when the size of the current block may be less than or equal to the threshold, the current block can be decoded in operation 815 based on one or more second motion vectors from the second reference motion vector candidate library.

[0176] According to an embodiment, when performing operations 825-835, a flag that can be associated with the current block can be used, indicating that a motion vector from a first reference motion vector candidate library or a second reference motion vector candidate library will be used.

[0177] Although Figure 8 The example box for process 800 is shown, but in some implementations, process 800 may include... Figure 8 The boxes depicted in the diagram are compared to additional boxes, fewer boxes, different boxes, or boxes with different arrangements. Alternatively or concurrently, two or more boxes of process 800 may be executed in parallel.

[0178] Furthermore, the proposed methods can be implemented using processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium to perform one or more of the proposed methods.

[0179] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 9 A computer system 900 is shown, which is suitable for implementing certain embodiments of the disclosed subject matter.

[0180] The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by the computer's central processing unit (CPU), graphics processing unit (GPU), or through decoding, microcode, or other means.

[0181] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0182] Figure 9 The components shown for the computer system 900 are exemplary in nature and are not intended to limit the scope or functionality of the computer software used to implement embodiments of this disclosure. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiments of the computer system 900.

[0183] Computer system 900 may include certain human-machine interface (HMI) input devices. Such HMI input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The HMI device may also be used to capture certain media, which need not be directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0184] Human-machine interface input devices may include one or more of the following: keyboard 901, mouse 902, touchpad 903, touch screen 910, data glove 1204, joystick 905, microphone 906, scanner 907, and camera 908 (only one of each is shown).

[0185] Computer system 900 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. These HMI output devices may include tactile output devices (e.g., tactile feedback via touchscreen 910, data glove 1204, or joystick 905, but may also include tactile feedback devices not used as input devices), audio output devices (e.g., speaker 909, headphones (not shown)), visual output devices (e.g., screen 910, including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which may output two-dimensional or higher-dimensional visual outputs through means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0186] The computer system 900 may also include human-accessible storage devices and related media, such as optical media including high-density read-only / rewritable optical discs (CD / DVD ROM / RW) 920 or similar media 921, thumb drives 922, removable hard disk drives or solid-state drives 923, conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated devices based on ROM / ASIC / PLD such as security software protectors (not shown), and so on.

[0187] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0188] Computer system 900 may also include interfaces to one or more communication networks 955. For example, network 955 may be wireless, wired, or optical. Network 955 may also be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), vehicular and industrial network, real-time network, latency-tolerant network, etc. Network 955 also includes Ethernet, wireless LAN, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial broadcast TV), vehicular and industrial networks (including CANBus), etc. Some networks 955 typically require an external network interface adapter 954 for connection to certain general-purpose data ports or peripheral buses 949 (e.g., the USB port of computer system 900); other systems are typically integrated into the core of computer system 900 via a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks 955, computer system 900 can communicate with other entities. The communication can be unidirectional, used only for receiving (e.g., wireless television), unidirectional, used only for sending (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as through a local area or wide area digital network to other computer systems. Each of the above-described network 955 and network interface 954 can use certain protocols and protocol stacks.

[0189] The aforementioned human-machine interface device, human-accessible storage device, and network interface can be connected to the core 940 of the computer system 900.

[0190] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 943, task-specific hardware accelerators 944, etc. These devices, along with read-only memory (ROM) 945, random access memory (RAM) 946, internal mass storage (e.g., internal non-user-accessible hard disk drives, solid-state drives (SSDs), etc.) 947, can be connected via the system bus 1248. In some computer systems, the system bus 1248 may be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus 1248 or connected via a peripheral bus 949. Peripheral bus architectures include External Peripheral Component Interconnect (PCI), Universal Serial Bus (USB), etc.

[0191] The CPU 941, GPU 942, FPGA 943, and accelerator 944 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 945 or RAM 946. Transient data can also be stored in RAM 946, while permanent data can be stored, for example, in internal mass storage 947. Fast storage and retrieval of any memory device can be achieved through the use of a cache memory, which can be closely associated with one or more CPUs 941, GPUs 942, mass storage 947, ROM 945, RAM 946, etc.

[0192] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of this disclosure, or they may be media and code well-known and usable by those skilled in the art of computer software.

[0193] By way of example and not limitation, a computer system having architecture 900, particularly core 940, can provide functionality as a processor (including CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the non-volatile core 940, such as core-internal mass storage 947 or ROM 945. Software implementing various embodiments of this disclosure can be stored in such a device and executed by core 940. Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause core 940, particularly the processor therein (including CPU, GPU, FPGA, etc.), to execute a specific process or a specific portion of a specific process described herein, including defining data structures stored in RAM 946 and modifying such data structures according to a software-defined process. Alternatively or as an alternative, the computer system can provide functionality that is logically hardwired or otherwise included in circuitry (e.g., accelerator 944), which can replace or operate with the software to execute the specific process or a specific portion of a specific process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing the software (such as an integrated circuit (IC)), circuitry containing the logic to perform the execution, or both. This disclosure includes any suitable combination of hardware and software.

[0194] While this disclosure describes several exemplary embodiments, any variations, arrangements, and various alternative equivalents fall within the scope of this disclosure. It will be understood that those skilled in the art will be able to design numerous systems and methods based on this application, which, while not expressly shown or described herein, are within the spirit and scope of this disclosure as long as they embody the principles of this disclosure.

Claims

1. A method for decoding a video bitstream, characterized in that, The method includes: Retrieve multiple motion vector prediction values ​​associated with the current block from a first reference motion vector candidate library. The retrieved multiple motion vector prediction values ​​are associated with multiple decoded blocks, and the multiple decoded blocks belong to the same superblock as the current block. The motion vector associated with the current block is determined based on the retrieved multiple motion vector prediction values; The first reference motion vector candidate library is updated at the coded block level by inserting the motion vector associated with the current block into the first reference motion vector candidate library; after decoding all blocks at a predefined level greater than the coded block or in a predefined region surrounding the current block, the second reference motion vector candidate library is updated at a predefined level greater than the coded block or in a predefined region surrounding the current block by inserting the motion vector associated with the current block into the second reference motion vector candidate library; and The current block is decoded based on the determined motion vector associated with it.

2. The method according to claim 1, characterized in that, The method further includes: Decoding of another block in the same superblock is based on the motion vector associated with the current block, which is inserted into the first reference motion vector candidate library.

3. The method according to claim 1, characterized in that, After decoding all blocks at a predefined level greater than the coded block, updating the second reference motion vector candidate library at the predefined level greater than the coded block by inserting the motion vector associated with the current block into the second reference motion vector candidate library includes: After decoding all blocks in the same superblock, the motion vector associated with the current block is inserted into the second reference motion vector candidate library to update the second reference motion vector candidate library at the superblock level.

4. The method according to claim 1, characterized in that, After decoding all blocks at a predefined level greater than the coded block, updating the second reference motion vector candidate library at the predefined level greater than the coded block by inserting the motion vector associated with the current block into the second reference motion vector candidate library includes: After decoding all blocks at a predefined level of the quadtree associated with the current block, the motion vector associated with the current block is inserted into the second reference motion vector candidate library to update the second reference motion vector candidate library at the predefined level of the quadtree.

5. The method according to claim 4, characterized in that, The predefined level of the quadtree is the first level of the quadtree after the superblock level.

6. The method according to any one of claims 1 to 5, characterized in that, Further includes: When the size of the current block being decoded is greater than a threshold, the current block is decoded based on multiple first motion vectors in the first reference motion vector candidate library; or When the size of the current block is less than or equal to the threshold, the current block is decoded based on multiple second motion vectors in the second reference motion vector candidate library.

7. The method according to any one of claims 1 to 5, characterized in that, Further includes: The flag associated with the current block is decoded, indicating that a motion vector from the first reference motion vector candidate library or the second reference motion vector candidate library will be used.

8. The method according to any one of claims 1 to 5, characterized in that, The retrieved multiple motion vector prediction values ​​are retrieved based on the recovered reference order.

9. The method according to claim 8, characterized in that, The recovery reference order is used to retrieve the plurality of motion vector prediction values ​​associated with the current block from the first reference motion vector candidate library based on satisfied conditions.

10. The method according to claim 9, characterized in that, The conditions are based on one of the following: The size of the current block; Quantize parameter values; or Hash hit rate of screen content.

11. A device for decoding video bitstreams, characterized in that, The device includes: At least one memory, the at least one memory being configured to store program code; and At least one processor, the at least one processor being configured to read the program code and perform the method as indicated by the program code as claimed in any one of claims 1 to 10.

12. A method for encoding a video stream, used to generate a video stream, characterized in that, include: Retrieve multiple motion vector predictions associated with the current block from a reference motion vector candidate library. The retrieved multiple motion vector predictions are associated with multiple encoded blocks, and the multiple encoded blocks belong to the same superblock as the current block. The motion vector associated with the current block is determined based on the retrieved multiple motion vector prediction values; The reference motion vector candidate library is updated at the coded block level by inserting the motion vector associated with the current block into the reference motion vector candidate library; after decoding all blocks at a predefined level greater than the coded block or in a predefined region surrounding the current block, the other reference motion vector candidate library is updated at a predefined level greater than the coded block or in a predefined region surrounding the current block by inserting the motion vector associated with the current block into another reference motion vector candidate library; and The current block is encoded based on the determined motion vector associated with it.

13. An apparatus for encoding a video stream, used to generate a video stream, characterized in that, The device includes: At least one memory, the at least one memory being configured to store program code; and At least one processor, the at least one processor being configured to read the program code and execute the method as instructed by the program code as claimed in claim 12.

14. A non-volatile computer-readable medium storing instructions, characterized in that, The instructions include one or more instructions that, when executed by one or more processors of the device, implement the method as described in any one of claims 1 to 10, 12.

15. A computer program product, comprising a computer program; characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 10 and 12.

16. A method for storing video streams, characterized in that, include: The method according to claim 12 generates a video stream; and The video stream is stored on a non-volatile computer-readable storage medium.

17. A method for storing video streams, characterized in that, include: Storing video streams on non-volatile computer-readable storage media; and The video stream is decoded using the method described in any one of claims 1 to 10.

Images