Method and apparatus for video coding and decoding using historical motion vector prediction table based on codec tree units

Abstract

The invention relates to a video coding and decoding method using a coding and decoding tree unit or a plurality of HMVP (History-Based Motion Vector Prediction) tables based on a plurality of History-Based Motion Vector Prediction). According to a method, a block in a current codec tree unit is encoded or decoded using information including a merge list or an adaptive motion vector prediction (AMVP) list, including one or more candidates from one or more HMVP tables based on codec tree units. The HMVP table based on codec tree units is maintained and updated on the basis of codec tree units. According to another method, if the location to be referenced is not within the first region, the location to be referenced is mapped to a mapping location of the first region and then the corresponding motion is referenced. If the mapped location of the first region does not move, a predefined default motion or a proximity motion at a proximity location is used as a corresponding motion.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application and claims priority to U.S. Provisional Patent Application No. 63 / 368,380 filed on July 14, 2022 and U.S. Provisional Patent Application No. 63 / 486,488 filed on February 23, 2023. The entire contents of the U.S. Provisional Patent Applications are hereby incorporated by reference. [Technical field]

[0003] The present invention relates to inter-frame prediction of video codecs, and more particularly to inter-frame prediction using a historical motion vector prediction table based on a codec tree unit. [Background technology]

[0004] Versatile Video Coding (VVC) is the latest international video codec standard developed by the Video Coding Experts Group (VCEG) of the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) and the Joint Video Experts Team (JVET) of the International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Moving Picture Experts Group (MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology - Coded representation of immersive media - Part 3: Versatile video codec, published in February 2021. VVC is developed on the basis of its predecessor High Efficiency Video Coding (HEVC), by adding more codec tools to improve codec efficiency and handle various types of video sources including three-dimensional (3D) video signals.

[0005] Figure 1AAn exemplary adaptive intra / inter video codec system with in-loop processing is shown. For intra prediction, prediction data is derived based on previously encoded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed on the encoder side, and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also called a residual. The prediction error is then processed by transform (T) 118, followed by quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 for inclusion in a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with side information, such as motion and codec modes associated with intra and inter prediction, and other information such as parameters of loop filters applied to the underlying image area. Side information related to intra prediction 110, inter prediction 112 and loop filter 130, such as Figure 1A As shown, the reference picture or pictures must also be reconstructed at the encoder end when using inter-frame prediction mode. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in the reference picture buffer 134 and used for prediction of other frames.

[0006] like Figure 1A As shown, the incoming video data undergoes a series of processes in the encoding system. The reconstructed video data from REC 128 may be subject to various impairments due to the series of processes. Therefore, a loop filter 130 is typically applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve the video quality. For example, a deblocking filter (DF), sample adaptive offset (SAO), and adaptive loop filter (ALF) may be used. The loop filter information may need to be included in the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 for inclusion in the bitstream. Figure 1A In FIG. 1 , the reconstructed samples are stored in a reference picture buffer 134 before the loop filter 130 is applied to reconstruct the video. Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to the High Efficiency Video Codec (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1BThe decoder shown may use the same or partially the same functional blocks as the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses an entropy decoder 140 instead of the entropy encoder 122 to decode the video bitstream into quantized transform coefficients and required codec information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 at the decoder side does not need to perform a pattern search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from the entropy decoder 140, without performing motion estimation.

[0008] According to VVC, the input picture is divided into non-overlapping square block areas, called codec tree units (codec tree units), similar to HEVC. Each codec tree unit can be divided into one or more coding units (CUs) of smaller size. The resulting CU partition can be square or rectangular. In addition, VVC divides the codec tree unit into prediction units (PUs), which are units for applying prediction processes (such as inter-frame prediction, intra-frame prediction, etc.).

[0009] The VVC standard incorporates a history-based merge mode invention, which is reviewed below.

[0010] History-based merge mode construction

[0011] History-based merge mode stores merge candidates of some previous CUs in a history array. For the current CU, in addition to the original merge mode candidate construction, it can also enrich the merge mode candidates with one or more candidates in the history array. The details of history-based motion vector prediction can be found in JVET-K0104 (Li Zhang et al., “CE4-related: History-based Motion Vector Prediction”, International Telecommunication Union - Telecommunication Standardization Sector (ITU-T) SG 16WP 3 and International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) JTC 1 / SC 29 / WG11, 11th Meeting: Ljubljana, Slovenia, July 10-18, 2018, Document: JVET-K0104).

[0012] In HMVP, a HMVP candidate table is maintained and updated in real time. After decoding a non-affine inter-coded block, the table is updated by adding the associated motion information as a new HMVP candidate to the last entry of the table. A first-in-first-out (FIFO) or constrained FIFO rule is applied to remove and add entries in the table. HMVP candidates can be applied to the merge candidate list or the AMVP candidate list.

[0013] A history-based MVP (HMVP) method is proposed, in which the HMVP candidate is defined as the motion information of the previously coded block. A table containing multiple HMVP candidates is maintained during the encoding / decoding process. When a new slice is encountered, the table is cleared. Whenever there is an inter-coded block, the associated motion information is added to the last entry of the table as a new HMVP candidate. The overall encoding and decoding process is as follows Figure 2 As shown, step 210 is performed to load the table with the initial HMVP candidate and update the table with the decoded motion information obtained from step 220 .

[0014] In the case where the table size S is set to 16, up to 16 HMVP candidates can be added to the table. If a previously coded block has more than 16 HMVP candidates, a First-In-First-Out (FIFO) rule is applied so that the table always contains the latest previously coded 16 motion candidates. Figure 3A An example is shown in which the FIFO rule is applied to remove a HMVP candidate and add a new one in the table used in the proposed method.

[0015] In order to further improve the encoding and decoding efficiency, a restricted first-in-first-out (FIFO) rule is introduced, that is, when inserting HMVP into the table, a redundancy check is first applied to find whether there is an identical HMVP candidate in the table. If found, the identical HMVP candidate is removed from the table, and all subsequent HMVP candidates are moved, that is, the index is reduced by 1. Figure 3B An example of restricted FIFO rule is shown, where candidate NMVP2 is found to be redundant and is removed after the update.

[0016] HMVP candidates can be used in the construction process of the merge candidate list. All HMVP candidates from the last entry to the first entry in the table are inserted after the temporal motion vector prediction (Tempory MVP) candidate. HMVP candidates are pruned. Once the total number of available merge candidates reaches the maximum allowed number of merge candidates for the signal, the construction process of the merge candidate list is terminated.

[0017] Similarly, HMVP candidates can also be used in the construction process of the AMVP candidate list. The motion vectors of the last K HMVP candidates in the table are inserted after the TMVP candidates. Only HMVP candidates with the same reference picture as the AMVP target reference picture are used to construct the AMVP candidate list. HMVP candidates are pruned. In JVET-K0104, K is set to 4.

[0018] In addition, when the total number of merge candidates is greater than or equal to 15, a truncated unary plus fixed length (3 bits) binarization method is applied to encode the merge index. The total number of merge candidates is denoted as N mrg ,The binarization methods are listed in Table 1.

[0019] Table 1 - Binary string of merged index (assuming N mrg for 15)

[0020]

[0021] During the development of the VVC standard, a coding tool called Non-Adjacent Motion Vector Prediction (NAMVP) was proposed in JVET-L0399 (Yu Han et al., "CE4.4.6: Improvements in Merge / Skip Mode", ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 of the Joint Video Exploration Team (JVET), 12th Meeting: Macau, China, October 3-12, 2018, Document: JVET-L0399). According to the NAMVP technique, non-adjacent spatial merge candidates are inserted after the TMVP (i.e., temporal MVP) in the regular merge candidate list. The mode of the spatial merge candidates is as follows: Figure 4 The distance between non-adjacent spatial candidates and the current codec block is based on the width and height of the current codec block. Figure 4 In , each small square corresponds to a NAMVP candidate, and the candidates are sorted by distance (as shown by the numbers inside the squares). No line buffer restrictions apply. In other words, NAMVP candidates far away from the current block may need to be stored, which may require a larger buffer.

[0022] In the present invention, a method and apparatus for video coding and decoding using a history-based MVP (motion vector prediction) table based on a codec tree unit is disclosed to improve performance. [Summary of the invention]

[0023] The present invention discloses a video coding method using multiple history-based motion vector prediction (MVP) tables. According to the method at the decoder end, coded data related to a current codec tree unit (codec tree unit) to be decoded is received. Blocks in the current codec tree unit are decoded using information including a merge list or an adaptive motion vector prediction (Adaptive Notion Vector Prediction, AMVP) list, wherein the merge list or the AMVP list includes one or more first candidates from one or more codec tree unit-based history MVP (HMVP) tables. The one or more codec tree unit-based HMVP tables are updated by storing the motion decoded for the current codec tree unit in one of the one or more codec tree unit-based HMVP tables to generate one or more updated codec tree unit-based HMVP tables. The merge list or the AMVP list is updated according to update information including one or more second candidates from the one or more updated codec tree unit-based HMVP tables.

[0024] Corresponding to the encoder side, pixel data related to the current codec tree unit is received. The motion of the block in the current codec tree unit is derived. The block in the current codec tree unit is encoded using information including a merge list or an adaptive motion vector prediction list, wherein the merge list or the adaptive motion vector prediction list includes one or more first candidates from one or more codec tree unit-based historical MVP tables. The one or more codec tree unit-based HMVP tables are updated by storing the motion derived for the current codec tree unit in one of the one or more codec tree unit-based HMVP tables to generate one or more updated codec tree unit-based HMVP tables. The merge list or the adaptive motion vector prediction list is updated according to the update information including one or more second candidates from the one or more updated codec tree unit-based HMVP tables.

[0025] In one embodiment, the corresponding motion of each predefined region is stored in one of the one or more codec tree unit based HMVP tables. In one embodiment, each predefined region corresponds to an 8x8 or 16x16 block.

[0026] In one embodiment, the target motion is stored in one of the one or more codec tree unit based HMVP tables after N blocks are decoded, where N is a positive integer. In another embodiment, the target motion is stored in the one or more codec tree unit based HMVP tables only when the target motion is far away from the motion previously stored in one of the one or more codec tree unit based HMVP tables. In another embodiment, the target motion of a corresponding block is stored in one of the one or more codec tree unit based HMVP tables only when the horizontal or vertical position of the corresponding block is on an MxM grid, where M is a positive integer.

[0027] In one embodiment, one or more second candidates from different codec tree unit-based HMVP tables are inserted into the merge list or AMVP list according to predefined positions and candidate positions in different codec tree unit-based HMVP tables. The predefined position may be determined according to the block width and block height of the block in the current codec tree unit.

[0028] According to another method, a first region of a current codec tree unit including a current block is selected to derive one or more first non-adjacent motion vector predictions. If the position to be referenced is not within the first region, the position to be referenced is mapped to a mapping position of the first region and then the corresponding motion is referenced. If there is no motion at the mapping position of the first region, a predefined default motion or a neighboring motion at a neighboring position is used as the corresponding motion.

[0029] In one embodiment, the neighboring position corresponds to the left 4x4 block, the right 4x4 block, the top 4x4 block or the bottom 4x4 block of the mapping position, or the first left or right 4x4 block with motion information of the mapping position.

Brief Description of the Drawings

[0030] Figure 1A An exemplary adaptive intra / inter video coding and decoding system including loop processing is described.

[0031] Figure 1B Describes Figure 1A The corresponding decoder of the encoder in .

[0032] Figure 2 An exemplary decoder flow is described that includes a history-based motion vector prediction candidate list.

[0033] Figure 3A An example of updating the HMVP table using a first-in-first-out (FIFO) structure is described.

[0034] Figure 3B An example of updating the HMVP table using a constrained first-in-first-out (FIFO) structure is described.

[0035] Figure 4 Exemplary patterns of non-adjacent spatial merging candidates are described.

[0036] Figure 5 An example is described for mapping motion information of a position to be referenced in a non-available area to a predefined position, wherein the predefined position is located in a row above a first row of coding tree units.

[0037] Figure 6 An example is described for mapping motion information of a position to be referenced in a non-available area to a predefined position, wherein the predefined position is located at a bottom line of a corresponding codec tree unit row.

[0038] Figure 7 An example is described for mapping motion information of a position to be referenced in a non-available area to a predefined position, wherein the predefined position is located at a bottom line or a center line of a corresponding codec tree unit row.

[0039] Figure 8 An example is described for mapping motion information of a position to be referenced in a non-available area to a predefined position, wherein the predefined position is located at a bottom line of a corresponding codec tree unit row or a codec tree unit row above the corresponding codec tree unit row.

[0040] Fig. 9 A flow chart of an exemplary video decoding system including a history-based MVP table based on a codec tree unit according to an embodiment of the present invention is described.

[0041] Fig.10 A flow chart of an exemplary video coding system including a history-based MVP table based on a codec tree unit according to an embodiment of the present invention is described.

[0042] Fig.11 A flow chart of another exemplary video decoding system including non-contiguous history-based MVP tables according to an embodiment of the present invention is described.

[0043] Fig.12 A flow chart of another exemplary video coding system including non-contiguous history-based MVP tables according to an embodiment of the present invention is described. [Specific implementation method]

[0044] It can be easily understood that the components of the present invention, as shown and described in the figures, can be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the system and method of the present invention, as shown in the figures, does not limit the scope of the present invention as described in the claims, but only represents selected embodiments of the present invention. References to "one embodiment", "one example" or similar language in this specification mean that specific features, structures or characteristics related to the embodiment can be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in one example" appearing in various places in this specification do not necessarily refer to the same embodiment.

[0045] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the relevant art will recognize that the present invention may be implemented without one or more specific details, or using other methods, components, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the present invention. Embodiments of the present invention may be best understood by reference to the drawings, in which like parts are represented by like numbers. The following description is by way of example only and briefly describes certain selected apparatus and method embodiments consistent with the claims described herein.

[0046] In order to improve the encoding and decoding efficiency of HMVP, some methods are disclosed to add more HMVP tables to increase the diversity of HMVP candidates.

[0047] Method 1: Complex HMVP Table

[0048] In one embodiment, it is recommended to use more than one HMVP table, which is generated by applying different update rules, such as different update frequencies. For example, a lookup table LUT-0 is updated once for each CU. Another lookup table LUT-1 is updated once within 5 CUs. In the present disclosure, the HMVP table is also referred to as a lookup table because the lookup table can be used to implement the HMVP table. According to another embodiment, the update rule may be related to the partition result associated with the CU, such as the quadtree (QT) depth, the binary tree (BT) depth, or the number of partitions associated with the CU. For example, if the QT depth / BT depth / partition time of the current block is less than 3, LUT-0 is updated. If the QT depth / BT depth / partition time of the current block is greater than 3, LUT-1 is updated.

[0049] In another embodiment, it is proposed to use more than one HMVP table, which is generated based on the difference between the motion to be added and the motion stored in the lookup table LUT, which is called MVD. For example, if the absolute value of the MVD between the motion to be added and any other motion in LUT-0 is greater than a threshold, such as 0, then LUT-0 is updated with a motion vector. If the absolute value of the MVD between the motion to be added and any other candidate in the lookup table LUT-1 is greater than another threshold, such as 32, then the lookup table LUT-1 is updated with a motion vector.

[0050] In another embodiment, it is suggested to use more than one HMVP table, which are generated based on the position of the corresponding CU. For example, the lookup table LUT-0 is updated only when the upper left position of the position to be inserted into the CU is in the 128x128 grid. The lookup table LUT-1 is updated only when the upper left position of the position to be inserted into the CU is in the 64x64 grid.

[0051] In another embodiment, it is recommended to store not only the motion information in the lookup table LUT, but also the position of the current CU. By doing so, more than one HMVP table can be created based on the position of the current CU. In one embodiment, the horizontal distance or vertical distance between the CU to be inserted and any CU storing motion information in the HMVP table can be used to determine whether to insert the motion information. For example, if the horizontal distance or vertical distance between the CU to be inserted and any CU storing motion information in the lookup table LUT-0 is greater than a threshold, such as 16, then LUT-0 is updated with motion information (e.g., a motion vector). If the horizontal distance or vertical distance between the CU to be inserted and any CU storing motion information in the lookup table LUT-1 is greater than another threshold, such as 64, then the lookup table LUT-1 is updated with motion information (e.g., a motion vector).

[0052] In another embodiment, it is suggested to create more than one HMVP table according to the symbol value of MVx and MVy. For example, 4 HMVP tables are created: LUT-0 is used to store motion vectors with symbol (MVx)>=0 and symbol (MVy)>=0; LUT-1 is used to store motion vectors with symbol (MVx)<0 and symbol (MVy)>=0; LUT-2 is used to store motion vectors with symbol (MVx)>=0 and symbol (MVy)>0; LUT-3 is used to store motion vectors with symbol (MVx)<0 and symbol (MVy)<0. For example, 8 HMVP tables are created for 8 pairs of symbols (MVx, Mvy).

[0053] In another embodiment, it is suggested to create more than one HMVP table according to the prediction mode of the CU. For example, two HMVP tables are created: LUT-0 is used to store motion vectors from the merge mode, and LUT-1 is used to store motion vectors from the non-merge mode.

[0054] In addition, the above embodiment can be further restricted so that if one lookup table LUT is updated, other lookup tables LUT cannot be updated. In other words, one motion information is used to update only one lookup table LUT.

[0055] In addition, the above embodiments can be further combined. For example, LUT-0 is updated with the CU in the 128x128 grid, and if the motion is different from any other motion in the lookup table LUT-0, the motion is inserted. Lookup table LUT-1 is updated with the CU in the 128x128 grid, and if the MVD between the motion information to be inserted and any other motion information in the lookup table LUT-1 is greater than a threshold, such as 64, the motion is inserted.

[0056] In another embodiment, a spatial domain multi-HMVP table may be generated. For example, a lookup table LUT is updated within N codec tree units. That is, in this LUT, only the motion information in these N codec tree units can be used to update the lookup table LUT. N can be any positive integer greater than 0. In this way, motion information across codec tree units / across codec tree unit rows may be used by referencing the spatial domain multi-HMVP table. In addition, the LUT may be further restricted to retain only the above-mentioned M codec tree unit rows.

[0057] Method 2: Insert candidates from the subsidiary HMVP table into the merged candidate list or AMVP MVP list

[0058] According to this method, N candidates in more than one HMVP table can be selected to be inserted into the merge candidate list or the AMVP MVP list. N can be any integer greater than 0.

[0059] In one embodiment, the historical motion vector prediction lookup tables (HMVP LUTs) store not only the motion information but also the upper left position of the coding unit (CU) to be inserted. Then, N candidates are selected based on the position of the CU. For example, the motion information closest to the current CU position will be selected before the motion information farther from the current CU position. In another embodiment, the motion information farther from the current CU position will be selected before the motion information close to the current CU position.

[0060] In another embodiment, N candidates are selected based on the distances between the motion information of the current CU and the corresponding CUs stored in the lookup table. These distances are designed according to the width and height of the current CU. For example, the distance between the motion information of the current CU and the corresponding CUs stored in the lookup table that is greater than the CU width or height and less than twice the CU width or height will be inserted first. After that, the distance between the motion information of the current CU and the corresponding CUs stored in the lookup table that is greater than twice the CU width or height and less than three times the CU width or height will be inserted.

[0061] In one embodiment, N additional historical motion vector prediction lookup tables are used, wherein candidates in the M lookup tables are added in order from oldest to newest, and candidates in the (NM) lookup tables are added in order from newest to oldest.

[0062] In one embodiment, more than one historical motion vector prediction lookup table is used. Candidates are added in an interleaved manner. For example, the newest motion in lookup table LUT-0 is added first. Then, the newest motion in lookup table LUT-1 is added. Next, the newest motion in lookup table LUT-2 is added. After that, the second newest motion in lookup table LUT-0 is added. Then, the second newest motion in lookup table LUT-1 is added. Next, the second newest motion in lookup table LUT-2 is added. Another example is to first insert K candidates from promising lookup table LUT-A, and then add motions from other lookup tables in an interleaved manner.

[0063] In another embodiment, more than one lookup table is used. The order in which the lookup tables are added is designed according to the size of the current CU. For example, 3 lookup tables are used. Lookup table LUT-0 is updated by CU motion in a 16x16 grid. Lookup table LUT-1 is updated by CU motion in a 64x64 grid. When the position of the current CU is in a 16x16 grid, the candidates in lookup table LUT-0 will be inserted before the candidates in lookup table LUT-1.

[0064] The inter-frame prediction based on multiple historical motion vector prediction methods proposed above can be combined with each other. For example, method 1 can be used together with method 2.

[0065] Approach 3: Limiting the available area for non-adjacent spatial merging candidates

[0066] Method 3 proposes a method to further reduce the bandwidth of supporting non-adjacent spatial merge candidates. In one embodiment, only the motion information in the current codec tree unit (CU) can be referenced by non-adjacent spatial merge candidates. In another embodiment, only the motion information in the current codec tree unit or the M codec tree units to the left can be referenced by non-adjacent spatial merge candidates. M can be any integer greater than 0. In another embodiment, only the motion information in the current codec tree unit row can be referenced by non-adjacent spatial merge candidates. In one embodiment, only the to-be-referenced positions within the current codec tree unit row or within the N codec tree unit rows above can be referenced. N can be any integer greater than 0.

[0067] In another embodiment, the motion information in the current codec tree unit, the current codec tree unit row, the current codec tree unit row + N upper codec tree unit rows, the current codec tree unit + M codec tree units on the left, or the current codec tree unit + N upper codec tree unit rows + M codec tree units on the left can be referenced without restriction. In addition, the motion information in other areas can only be referenced through larger predefined units. For example, the motion information in the current codec tree unit row is stored in a 4x4 grid, while other motion information outside the current codec tree unit row is stored in a 16x16 grid. In other words, a 16x16 area only needs to store one motion information, so the position to be referenced should be rounded to the 16x16 grid, or changed to the nearest position of the 16x16 grid.

[0068] In another embodiment, the motion information in the current codec tree unit row or the current codec tree unit row + M codec tree unit rows can be referenced without restriction, and for the positions to be referenced above the codec tree unit row, these positions will be mapped to a row above the current codec tree unit, or the current codec tree unit row + M codec tree unit rows for reference. This design can retain most of the coding efficiency and will not significantly increase the buffer due to storing the motion information of the upper codec tree unit rows. For example, the motion information in the current codec tree unit row (510) and the first upper codec tree unit row (512) can be referenced without restriction; and for the positions to be referenced in the second (520), third (522), fourth codec tree unit row, etc., these positions will be mapped to a row (530) above the first upper codec tree unit row before reference. Figure 5 , a dark circle represents an unavailable candidate 540, an empty circle represents an available candidate 542, and a dot-filled circle represents an unavailable candidate 544. For example, an unavailable candidate 550 in the second (520) codec tree unit row above is mapped to an available candidate 552 in a row (530) above the first codec tree unit row (524) above.

[0069] In the above example, the area that can be referenced without restriction is close to the current codec tree unit (for example, the current codec tree unit row or the first codec tree unit row above). However, the area according to the present invention is not limited to the area shown in the above example. The area can be larger or smaller than shown in the above example. Generally, the area can be limited to one or more predefined distances from the current codec tree unit in the vertical direction, horizontal direction, or both. In the above example, the area is limited to 1 codec tree unit height in the vertical direction above, and can be extended to 2 or 3 codec tree unit heights if necessary. In the case of using M codec tree units on the left, the limitation is M codec tree unit widths of the current codec tree unit row. The horizontal position of the position to be referenced and the horizontal position of the mapped predefined position can be the same (for example, position 550 and position 552 are at the same horizontal position). However, other horizontal positions can also be used.

[0070] In another embodiment, the motion information of the current codec tree unit row or the current codec tree unit row + M codec tree unit rows can be referenced without restriction. In addition, for the position to be referenced in the above codec tree unit row, the position will be mapped to the last row of the corresponding codec tree unit row for reference. For example, Figure 6 As shown, the motion information of the current codec tree unit row (510) and the first codec tree unit row (512) above can be referenced without restriction, and for the position to be referenced in the second codec tree unit row (520) above, the position will be mapped to the bottom line (610) of the second codec tree unit row (520) above before the reference. For the position to be referenced in the third codec tree unit row (522) above, the position will be mapped to the bottom line (620) of the third codec tree unit row (522) above before the reference. Figure 6 The legend of the candidate types (i.e., 540, 542, and 544) and Figure 5 The same as in.

[0071] In another embodiment, the motion information of the current codec tree unit row or the current codec tree unit row + M codec tree unit rows can be referenced without restriction, and for the position to be referenced in the upper codec tree unit row, according to the position of the motion information to be referenced, the position will be mapped to the last row, bottom line or center line of the corresponding codec tree unit row for reference. Figure 7As shown, the motion information of the current codec tree unit row (510) and the first codec tree unit row (512) above can be referenced without restriction. For the position 1 to be referenced in the second codec tree unit row (520) above, the position will be mapped to the bottom line (610) of the second codec tree unit row above before the reference. However, for the position 2 to be referenced in the second codec tree unit row above, since it is closer to the bottom line (610) than the center line (710), the position will be mapped to the center line (710) of the second codec tree unit row (520) above before the reference. Figure 7 The legend of the candidate types (i.e., 540, 542, and 544) and Figure 5 The same as in.

[0072] In another embodiment, the motion information of the current codec tree unit row or the current codec tree unit row + M codec tree unit rows can be referenced without restriction, and for the position to be referenced in the upper codec tree unit row, according to the position of the motion information to be referenced, the position will be mapped to the last row or bottom line of the corresponding codec tree unit row for reference. Figure 8 As shown, the motion information of the current codec tree unit row (510) and the first codec tree unit row (512) above can be referenced without restriction. For the position 1 to be referenced in the second codec tree unit row (520) above, the position will be mapped to the bottom line (610) of the second codec tree unit row (520) above before reference. However, for the position 2 to be referenced in the second codec tree unit row (520) above, since it is closer to the bottom line (610) of the second codec tree unit row above than to the bottom line (620) of the third codec tree unit row above, the position will be mapped to the bottom line (620) of the third codec tree unit row (522) above before reference. Figure 8 The legend of the candidate types (i.e., 540, 542, and 544) and Figure 5 The same as in.

[0073] In another embodiment, the motion information of the current codec tree unit or the current codec tree unit + N left codec tree units can be referenced without restriction, and for the left codec tree unit, the position to be referenced will be mapped to the rightmost line closest to the current codec tree unit or the current codec tree unit + N left codec tree units. For example, the motion information of the current codec tree unit and the first left codec tree unit can be referenced without restriction, and if the position to be referenced is in the second left codec tree unit, the position will be mapped to a line on the left side of the first left codec tree unit before reference. If the position to be referenced is in the third left codec tree unit, the position will be mapped to a line on the left side of the first left codec tree unit before reference. For example, the motion information of the current codec tree unit and the first left codec tree unit can be referenced without restriction, and if the position to be referenced is in the second left codec tree unit, the position will be mapped to the rightmost line of the second left codec tree unit before reference. If the position to be referenced is in the third left codec tree unit, the position will be mapped to the rightmost line of the third left codec tree unit before reference.

[0074] Method 4: Storing Space in Codec Tree Unit-based Lookup Tables (LUTs) - HMVP

[0075] In one embodiment, the HMVPs based on the codec tree unit are used to maintain the motion within each codec tree unit. That is, after decoding a codec tree unit, all the motions within the codec tree unit will be stored in an HMVP LUT based on the codec tree unit. For the current coding unit, it can reference different motions of different codec tree units from the corresponding HMVP LUTs based on the codec tree unit. For example, the motion of the upper codec tree unit, the upper right codec tree unit, the upper left codec tree unit, or the left codec tree unit can be referenced from different HMVP LUTs based on the codec tree unit.

[0076] In another embodiment, not every motion in a codec tree unit is reserved for codec tree unit based HMVPs. For example, only one motion is reserved within a predefined area (e.g., 8x8 or 16x16). Another example is that after decoding N coding units, one motion will be reserved for codec tree unit based HMVPs. Another example is that only when the position of the to-be-retained motion is far away from the positions of all previously decoded motions stored in the HMVP, it can be reserved for codec tree unit based HMVPs. Another example is that only when the to-be-inserted motion comes from a coding unit with an x-axis or y-axis position in an MxM grid, the motion will be reserved for codec tree unit based HMVPs. N and M in the previous examples can be any integer greater than zero.

[0077] In another embodiment, HMVPs from different codec tree units can be inserted based on a set of predefined positions. For example, the positions can be as follows: Figure 5 ). Another example is that the position is designed based on the width and height of the current coding unit. In another embodiment, the allowed reference motion area can be restricted to control the use of the motion buffer. For example, only the motion of the current codec tree unit row and the row above the current codec tree unit row can be referenced. Candidates from predefined positions of other codec tree unit rows will be mapped to the corresponding positions of the row above the current codec tree unit row before referencing. The allowed reference motion area can also be restricted by any other constraints mentioned in method 3.

[0078] The present disclosure proposes to store not only motion information but also the upper left and lower right positions of the CU (i.e., the position of the candidate) in the HMVP. Through this technology, it is possible to check whether the predefined position is covered by the motion stored in the HMVP. If the predefined position is to the right and below the upper left position of the HMVP and to the left and above the lower right position of the HMVP, the predefined position is covered by the CU of the HMVP. Therefore, candidates from historical motion vector predictions based on different codec tree units can be inserted into a merged or adaptive motion vector prediction list based on the predefined position and the position of the candidate in the historical motion vector prediction table based on different codec tree units.

[0079] The present disclosure proposes to store not only motion information but also CU in the center position of HMVP. Through this technology, it is possible to check whether the predefined position is covered by the motion stored in HMVP. If the predefined position is very close to the center position of HMVP, the predefined position is considered to be covered by the CU of HMVP.

[0080] In another embodiment, if the predefined reference location is not available, the motion of the CU covering the corresponding position stored in the spatial HMVP lookup table can be used. Otherwise, the predefined reference location will be mapped to the nearest allowed area. For example, the disallowed reference location may be a location outside the current codec tree unit row. The predefined reference locations in the upper codec tree unit row will be checked to determine whether they can be covered by any spatial HMVP. If they can be covered by the spatial HMVP, the motion of this corresponding spatial HMVP will be used for reference. Otherwise, they will be mapped to the bottom line of the first upper codec tree unit row.

[0081] As another example, if a predefined reference position is not available, the motion of the CU closest to the corresponding position stored in the spatial HMVP lookup table may be used.

[0082] For the spatial HMVP mentioned above, the candidates covered by the predefined area can be stored in the same lookup table. For example, the motion in a codec tree unit can be stored in a spatial HMVP lookup table. For another example, the motion in an NxM area can be stored in a lookup table, where N and M can be any integer greater than zero. NxM can be based on the resolution of the picture, the codec tree unit size, the decoding unit size, or the QP design.

[0083] Method 5: If there is no motion information at the mapped location, use a predefined motion

[0084] In one embodiment, in the above method, it is allowed to use a predefined area for reference. If the position to be referenced is not in the allowed area, it is mapped to the corresponding position of the allowed area before reference. If the mapping position has motion, it can be directly referenced. Otherwise, the mapping candidate becomes invalid. For example, the mapping block is encoded by an internal mode, an internal TMP mode, a MIP mode, a TIMD, a DIMD, an IBC mode, etc.

[0085] In order to increase the number of valid candidates, in one embodiment, a predefined default motion may be used, such as (MVx, Mvy) = (-cuWidth, -cuHeight), or (MVx, Mvy) = (-N*cuWidth, -M*cuHeight), or a MV designed based on the farthest position allowed to be referenced. In another embodiment, the motion of the neighboring position will be used, such as one of the 4x4 blocks to the left of the mapping position, one of the 4x4 blocks to the right of the mapping position, the first 4x4 block to the left containing motion information, or the first 4x4 block to the right containing motion information, or the first 4x4 block at the top containing motion information, or the first 4x4 block at the bottom containing motion information.

[0086] Any of the above proposed inter-frame prediction methods based on historical motion vector prediction based on codec tree units or historical motion vector prediction methods based on multiple non-adjacent motion vector predictions can be implemented in an encoder and / or decoder. For example, any of the proposed methods can be implemented in an inter-frame codec module (e.g. Figure 1A Inter-frame prediction 112 in the decoder) or the inter-frame coding module of the decoder (e.g. Figure 1B 152 in MC 152), or in a merge candidate list adaptive motion vector prediction candidate list derivation module of an encoder or decoder. Alternatively, any proposed method can be implemented as one or more circuits or processors, which are connected to the inter-frame / intra-frame / prediction / entropy coding and decoding module of the encoder and / or decoder to process data or provide information required by the inter-frame / intra-frame / prediction module.

[0087] Fig. 9A flowchart of an exemplary video decoding system is shown, which incorporates a codec tree unit-based history-based motion vector prediction table according to an embodiment of the present invention. The steps shown in the flowchart can be implemented as program code executable on one or more processors (e.g., one or more CPUs) at the encoder end. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to this method, in step 910, encoded data related to a current codec tree unit to be decoded is received. In step 920, a block in the current codec tree unit is decoded using information including a merge list or an adaptive motion vector prediction list, wherein the merge list or the adaptive motion vector prediction list includes one or more first candidates from one or more codec tree unit-based history motion vector prediction tables. In step 930, the one or more codec tree unit-based history motion vector prediction tables are updated by storing the motion decoded for the current codec tree unit in one of the one or more codec tree unit-based history motion vector prediction tables to generate one or more updated codec tree unit-based history motion vector prediction tables. In step 940 , the merge list or the adaptive motion vector prediction list is updated according to the update information including the one or more second candidates from the one or more updated codec tree unit based historical motion vector prediction tables.

[0088] Fig.10 A flow chart of an exemplary video coding system is shown, which incorporates a codec tree unit based history based motion vector prediction table according to an embodiment of the present invention. According to the method, pixel data associated with a current codec tree unit is received in step 1010. Motion of a block in the current codec tree unit is derived in step 1020. In step 1030, the block in the current codec tree unit is encoded using information including a merge list or an adaptive motion vector prediction list, wherein the merge list or the adaptive motion vector prediction list includes one or more first candidates from one or more codec tree unit based history motion vector prediction tables. In step 1040, the one or more codec tree unit based history motion vector prediction tables are updated by storing the motion derived for the current codec tree unit in one of the one or more codec tree unit based history motion vector prediction tables to generate one or more updated codec tree unit based history motion vector prediction tables. In step 1050, the merge list or the adaptive motion vector prediction list is updated according to the updated information including one or more second candidates from the one or more updated codec tree unit based history motion vector prediction tables.

[0089] Fig.11A flowchart of another exemplary video decoding system in combination with a non-adjacent history-based motion vector prediction table according to an embodiment of the present invention is shown. According to this method, in step 1110, coded data related to a current block to be decoded is received. In step 1120, one or more first non-adjacent motion vector prediction candidates are derived based on previously decoded motion information in a first region including a current codec tree unit, wherein the first region is limited to one or more predefined distances in a vertical direction, a horizontal direction, or both from the current codec tree unit, and if the position to be referenced is not in the first region, the position to be referenced is mapped to a mapping position in the first region and then the corresponding motion is referenced, and if the mapping position in the first region has no motion, a predefined default motion or a neighboring motion at a neighboring position is used as the corresponding motion. In step 1130, a merge candidate list including the one or more first non-adjacent motion vector prediction candidates is generated. In step 1140, current motion information of the current block is derived from the coded data according to the merge candidate list.

[0090] Fig.12 A flowchart of another exemplary video coding system in combination with a non-adjacent history-based motion vector prediction table according to an embodiment of the present invention is shown. In step 1210, pixel data related to a current block is received. In step 1220, current motion information of the current block is derived. In step 1230, one or more first non-adjacent motion vector prediction candidates are derived based on previously encoded motion information in a first region including a current codec tree unit, wherein the first region is limited to one or more predefined distances in a vertical direction, a horizontal direction, or both from the current codec tree unit, and if the position to be referenced is not in the first region, the position to be referenced is mapped to a mapping position in the first region and then the corresponding motion is referenced, and if the mapping position of the first region has no motion, a predefined default motion or a neighboring motion at a neighboring position is used as the corresponding motion. In step 1240, a merge candidate list including the one or more first non-adjacent motion vector prediction candidates is generated. In step 1250, the current motion information of the current block is encoded according to the merge candidate list.

[0091] The flowchart shown is intended to illustrate an example of video coding and decoding according to the present invention. A person familiar with the technology can modify each step, rearrange the steps, split the steps, or combine the steps to practice the present invention without departing from the spirit of the present invention. In this disclosure, specific syntax and semantics are used to illustrate examples of implementing the present invention. A person familiar with the technology can practice the present invention by replacing the syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention.

[0092] The above description is intended to enable a person skilled in the art to practice the invention in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, it is not intended that the present invention be limited to the specific embodiments shown and described, but rather to be given the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are shown in order to provide a deeper understanding of the present invention. However, a person skilled in the art will understand that the present invention may be practiced.

[0093] The embodiments of the present invention as described above can be implemented in various hardware, software codes, or a combination of the two. For example, an embodiment of the present invention can be one or more circuits integrated into a video compression chip, or integrated into video compression software to perform the program code described herein. An embodiment of the present invention can also be a program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The present invention may also involve multiple functions performed by a computer processor, a digital signal processor, a microprocessor, or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the invention, defining the specific methods embodied by the invention by executing machine-readable software code or firmware code. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles, and languages ​​of software codes and other means of configuring codes to perform tasks according to the invention will not depart from the spirit and scope of the invention.

[0094] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered in all respects only as illustrative and not restrictive. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All changes within the meaning and equivalent range of the claims should be included within their scope.

Claims

1. A video decoding method, the method comprising: Receiving, at a decoder, coded data associated with a current codec tree unit (CTU) to be decoded; decoding a block in the current codec tree unit using information including a merge list or an adaptive motion vector prediction (AMVP) list, wherein the merge list or the motion vector prediction list includes one or more first candidates from one or more historical motion vector prediction (HMVP) tables based on the codec tree unit; updating the one or more codec tree unit-based historical motion vector prediction tables by storing one of the motions decoded for the current codec tree unit in the one or more codec tree unit-based historical motion vector prediction tables to generate one or more updated codec tree unit-based historical motion vector prediction tables; as well as The merge list or the adaptive motion vector prediction list is updated according to update information including one or more second candidates from the one or more updated codec tree unit based historical motion vector prediction tables.

2. The method of claim 1, wherein the corresponding motion of each predetermined area of ​​the current codec tree unit is stored in the one or more codec tree unit-based historical motion vector prediction tables.

3. The method of claim 2, wherein each of the predefined areas corresponds to an 8x8 or 16x16 block.

4. The method of claim 1, wherein after N blocks are decoded, the target motion is stored in the one or more codec tree unit based historical motion vector prediction tables, and wherein N is a positive integer.

5. The method of claim 1, wherein a target motion is stored in one or more codec tree unit-based historical motion vector prediction tables only when the target motion is far away from a motion previously stored in one of the one or more codec tree unit-based historical motion vector prediction tables.

6. The method of claim 1 , wherein the target motion of the corresponding block is stored in one of the one or more codec tree unit-based historical motion vector prediction tables only when the horizontal or vertical position of the corresponding block is located on an MxM grid, and wherein M is a positive integer.

7. The method of claim 1, wherein one or more second candidates from different codec tree unit-based historical motion vector prediction tables are inserted into the merge list or the adaptive motion vector prediction list according to predefined positions and candidate positions in different codec tree unit-based historical motion vector prediction tables.

8. The method of claim 7, wherein the predefined position is determined according to a block width and a block height of a block in a current codec tree unit.

9. A video encoding method, the method comprising: Receiving pixel data associated with a current codec tree unit at an encoder end; Derived the motion of the block in the current codec tree unit; encoding a block in the current codec tree unit using information including a merge list or an adaptive motion vector prediction list, wherein the merge list or the adaptive motion vector prediction list includes one or more first candidates from one or more codec tree unit-based historical motion vector prediction tables; updating the one or more codec tree unit based historical motion vector prediction tables by storing the motion derived for the current codec tree unit in one of the one or more codec tree unit based historical motion vector prediction tables to generate one or more updated codec tree unit based historical motion vector prediction tables; as well as The merge list or the adaptive motion vector prediction list is updated according to update information including one or more second candidates from the one or more updated codec tree unit based historical motion vector prediction tables.

10. A video decoding method, the method comprising: Receiving coded data associated with a current block to be decoded at a decoder end; deriving one or more first non-neighboring motion vector prediction candidates based on previously decoded motion information in a first region including the current codec tree unit, wherein the first region is limited to one or more predefined distances in a vertical direction, a horizontal direction, or both from the current codec tree unit, and if a position to be referenced is not in the first region, mapping the position to be referenced to a mapping position in the first region and then referencing a corresponding motion, and if there is no motion in the mapping position of the first region, using a predefined default motion or a neighboring motion at a neighboring position as the corresponding motion; generating a merge candidate list including the one or more first non-adjacent motion vector prediction candidates; as well as The current motion information of the current block is derived from the encoded data according to the merge candidate list.

11. The method of claim 10, wherein the neighboring position corresponds to a left 4x4 block, a right 4x4 block, a top 4x4 block or a bottom 4x4 block of the mapped position, or a first left or right 4x4 block with motion information of the mapped position.

12. A video encoding method, the method comprising: Receiving pixel data associated with a current block at an encoder end; Derive the current motion information of the current block; deriving one or more first non-neighboring motion vector prediction candidates based on previously encoded motion information in a first region including the current codec tree unit, wherein the first region is limited to one or more predefined distances in a vertical direction, a horizontal direction, or both from the codec tree unit, and if a position to be referenced is not in the first region, mapping the position to be referenced to a mapping position in the first region and then referencing the corresponding motion, and if there is no motion in the mapping position of the first region, using a predefined default motion or a neighboring motion at a neighboring position as the corresponding motion; generating a merge candidate list including the one or more first non-adjacent motion vector prediction candidates; as well as The current motion information of the current block is encoded according to the merge candidate list.

13. The method of claim 12, wherein the neighboring position corresponds to a left 4x4 block, a right 4x4 block, a top 4x4 block or a bottom 4x4 block of the mapped position, or is the first left or right 4x4 block of the mapped position having motion information.

Images