Intra-frame merging mode derived from template-based intra-frame mode: method and apparatus

Abstract

A method and apparatus for video encoding and decoding are disclosed. According to the method, input data associated with the current block is received, wherein the input data includes pixel data to be encoded at the encoder end or encoded data associated with the current block to be decoded at the decoder end. Previously encoded / decoded blocks for the current block are determined, wherein the previously encoded / decoded blocks generated TIMD (Template-based Intra Mode Derivation) information or have stored TIMD information. Inherited TIMD information is derived from the previously encoded / decoded blocks, wherein the inherited TIMD information includes reference line information, wide-angle conditions, or both. Target prediction is generated for the current block using the inherited TIMD information. The current block is encoded or decoded using information including the target prediction.

Description

[0001] Cross-referencing This invention is a non-provisional application of U.S. Provisional Patent Application No. 63 / 625,332 (filed January 26, 2024), and claims priority thereto. The entire contents of that U.S. Provisional Patent Application are incorporated herein by reference. [Technical Field] This invention relates to video encoding and decoding systems. Specifically, this invention relates to inheritance information from previous codec blocks in video encoding and decoding systems, wherein the previous codec blocks generate template-based intra-mode derivation (TIMD) inheritance information and / or store TIMD inheritance information. [Background Technology] Versatile video coding (VVC) is the latest international video coding standard jointly developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Video Experts Group (JVET) of the ISO / IEC Moving Picture Experts Group (MPEG). This standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Informationtechnology - Coded representation of immersive media - Part 3: Versatile Video Coding, released in February 2021. VVC improves coding and decoding efficiency by adding more coding and decoding tools to its predecessor, HEVC (High Efficiency Video Coding), and can handle various types of video sources, including 3D video signals.

[0004] Figure 1AAn exemplary adaptive inter / intra video coding system incorporating loop processing is described. For intra prediction 110, prediction data is derived from previously encoded / decoded video data in the current frame. For inter prediction 112, motion estimation (ME) is performed on the encoder side, and motion compensation (MC) is performed based on the result of ME to provide prediction data derived from other frames and motion data. Switch 114 selects either intra prediction 110 or inter prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The residuals from transform and quantization are then encoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged together with side information (e.g., motion and encoding / decoding modes associated with intra-frame and inter-frame prediction) and other information (e.g., parameters associated with loop filters applied to the underlying image area). The side information associated with intra-frame prediction 110, inter-frame prediction 112, and the intra-loop filter 130 is provided to the entropy encoder 122, such as... Figure 1A As shown. When using inter-frame prediction mode, one or more reference images must also be reconstructed at the encoder end. Therefore, the transformed and quantized residuals are processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residuals. Then, at reconstruction (REC) 128, the residuals are added back to the prediction data 136 to reconstruct the video data. The reconstructed video data can be stored in the reference image buffer 134 and used to predict other frames.

[0005] like Figure 1AAs shown, the input video data undergoes a series of processing steps in the encoding system. Due to these processing steps, the reconstructed video data from REC 128 may suffer various forms of degradation. Therefore, a loop filter 130 is frequently applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve video quality. For example, a deblocking filter (DF), Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF) may be used. It may be necessary to incorporate the loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 for incorporation into the bitstream. Figure 1A In the process, the loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in the reference image buffer 134. Figure 1A The system described herein is intended to illustrate an exemplary architecture of a typical video encoder. It may correspond to the High Efficiency Video Codec (HEVC) system, VP8, VP9, ​​H.264, VVC, or any other video codec standard.

[0006] like Figure 1B As shown, the decoder can use similar or identical function blocks to the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and the required encoding / decoding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not need to perform mode search. Instead, the decoder only needs to generate intra-frame predictions based on the intra-frame prediction information received from entropy decoder 140. Furthermore, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from entropy decoder 140 without motion estimation.

[0007] I. Related Technologies I.1 Intra-mode encoding and decoding with 67 intra-prediction modes In order to capture arbitrary edge directions presented in natural video, the number of directional intra modes in VVC has been expanded from 33 used in HEVC to 65.

[0008] In VVC, several regular-angle intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes for non-square blocks.

[0009] I.2 Intra-frame mode encoding and decoding To reduce the complexity of generating the Most Probable Mode (MPM) list, an intra-frame mode encoding / decoding method with 6 MPMs (or primary MPMs) is adopted, considering two available adjacent intra-frame modes. The following three aspects are considered when constructing the MPM list: Default intra-frame mode Adjacent intra-frame mode Exported intra-frame mode I.3 Decoder-side Intra Mode Derivation (DIMD) When DIMD is applied, two intra-frame patterns are derived from reconstructed neighboring samples (templates), and these two predictors are combined with the planar pattern predictors, with weights derived from gradients. The DIMD pattern is used as an alternative prediction pattern and is always checked in the more complex RDO pattern.

[0010] To implicitly derive the intra-prediction mode of the block, texture gradient analysis is performed at both the encoder and decoder. This process begins with an empty Histogram of Gradients (HoG) with 65 entries, corresponding to 65 angular modes. The magnitudes of these entries are determined during texture gradient analysis.

[0011] Figure 2A -C shows an example of HoG calculated after applying the above operations to all pixel positions in the template. Figure 2A An example of template 220 selected for the current block 210 is shown. Template 220 includes T rows above the current block and T columns to the left of the current block. For intra-frame prediction of the current block, the regions 230 above and to the left of the current block correspond to the reconstruction region, and the regions 240 below and to the right of the block correspond to the unavailable region. Figure 2B An example with T=3 is shown, where HoG is calculated for pixel 260 in the middle row and pixel 262 in the middle column. For example, for pixel 252, a 3x3 window of 250 is used. Figure 2C An example of the amplitude (ampl) calculated for the angular intra-frame prediction mode is shown.

[0012] I.4 Template-based Intra Mode Derivation (TIMD) Template-based intra-mode derivation (TIMD) implicitly derives the intra-predictive mode of the CU in the encoder and decoder through neighboring templates, rather than communicating intra-predictive mode bits to the decoder. Predictive samples for the template are generated using template reference samples for each candidate mode. The cost is calculated as the SATD between the predicted and reconstructed samples of the template. The two intra-predictive modes with the minimum SATD are selected as TIMD modes. These two TIMD modes are weighted and fused after applying the PDPC procedure, and this weighted intra-predictive mode is used to encode and decode the current CU.

[0013] I.5 Extrapolation Filter-Based IntraPrediction (EIP) Mode The EIP schema is disclosed in JVET-AF0080. A brief overview of the EIP schema is as follows.

[0014] I.5.1 Obtaining the EIP Filter Three EIP filter shapes are proposed in this method.

[0015] I.5.1.1 Exporting EIP Coefficients (EIP Export Mode) The decoder decodes the relevant syntax elements to determine the type of the selected reconstructed region and the filter shape of the current block. The selected filter moves horizontally or vertically within the selected reconstructed region, constructing an autocorrelation matrix and cross-correlation vector with a one-pixel stride. The process of calculating coefficients from the autocorrelation matrix and cross-correlation vector is the same as in the convolutional cross-component model (CCCM).

[0016] I.5.1.2 EIP Filter Inheritance (EIP Combining Mode) The filter shape and filter coefficients can be inherited from previously decoded blocks with EIP or EIP merging modes. When the current block uses the EIP mode, the decoder decodes the EIP merging flag to determine whether to use the proposed merging mode. When the EIP merging flag is true, the merging index is further decoded. The EIP merging list includes spatially adjacent and non-adjacent candidates, temporal candidates, and historical candidates.

[0017] I.5.2 Prediction of the current block The EIP mode generates the prediction value for the current block by predicting diagonally from the top left to the bottom right position.

[0018] The predicted value in this contribution is calculated as follows: .

[0019] in It is the predicted value of the current block at (x, y). It is the i-th coefficient of the selected EIP filter, with coefficient indices ranging from 0 to 14. It is a reconstructed or predicted value used for the prediction of the current location. and These are the position offsets of the current position along the x and y directions, respectively.

[0020] I.6 Template-based Multi-Reference Line Intra-Frame Prediction Template-based multiple reference line intra-prediction (TMRL) mode combines reference lines and prediction modes, and uses template matching to construct a list of candidate combinations. The index of the candidate combination list is used to indicate which reference line and prediction mode to use when encoding and decoding the current block.

[0021] Since the extended reference line starts from reference line 1, the area covered by reference line 0 is used for template matching. For example... Figure 3 As shown, the SAD cost (i.e., TMRL cost) between prediction (generated from 50 combinations) and reconstruction is calculated over the template region. The 20 combinations with the lowest SAD cost are selected and arranged in ascending order to form a TMRL candidate list.

[0022] I.7 Inter-frame prediction The JVET-T2002 document describes the algorithms for Versatile Video Coding (VVC) and Test Model 11 (VTM 11). A brief description of VVC's inter-frame prediction is as follows.

[0023] I.7.1 Spatial Candidate Export Spatial merging candidates from Figure 4 Choose from the candidates shown.

[0024] I.7.2 Non-adjacent spatial candidates The patterns of non-adjacent spatial merging candidates are as follows: Figure 5 As shown. The distance between non-adjacent candidates and the current codec block is based on the width and height of the current codec block. Line buffer limits do not apply.

[0025] I.8 Intra-frame prediction using merged gradient histograms JVET-AG0078 introduces a novel intra-prediction mode called Merged Intra Mode Derivation (MIMD), based on the computation of a Merged Histogram of Gradients (MHoG). Similar to DIMD, up to five MIMD modes are derived from the MHoG and blended. The derivation process for modes and blending weights is the same as deriving DIMD modes and blending weights from the HoG. However, unlike DIMD, MHoG is not computed by directly analyzing template samples, but rather based on information extracted from neighboring blocks. More details can be found in JVET-AG0078.

[0026] Specifically, several N neighboring blocks are considered. If a neighboring block uses at least one directional intra-prediction mode, then that neighboring block is considered. This refers to the amplitude of the directional mode m in HoG, where m can take values ​​from 0 to... ,in This is the maximum number of intra-frame prediction modes. When considering... In such cases, a normalization process can be used.

[0027] Then, the MHoG is calculated using all HoGs extracted from available neighboring blocks. : , Finally, the MIMD modes and weights are calculated using MHoG. The five directional modes corresponding to the highest amplitudes in the MHoG and their weights are selected as the directional modes and weights for MIMD.

[0028] I.9 Occurrence-Based Intra Coding (OBIC) JVET-AG0078 discloses an Occurrence-Based Intra Coding (OBIC) method that derives the intra-prediction mode of the current block based on the sample-by-sample occurrence count of intra-modes within the current block's spatial neighborhood. To do this, adjacent and non-adjacent spatial neighborhood blocks are examined, and the intra-prediction modes of the blocks are collected into an occurrence histogram. Unlike the Histogram of Gradient (HoG) in DIMD, OBIC introduces an occurrence histogram (HoC), which includes the intra-mode and its sample-by-sample occurrence count. The occurrence value is calculated based on the number of samples encoded and decoded in that neighborhood using a particular intra-prediction mode. For example, if a uiWidth × uiHeight block is encoded and decoded in IPM mode, the occurrence count of that mode in that specific block is calculated as follows: HoC[IPM] += uiWidth * uiHeight; Where uiWidth and uiHeight are the width and height of the spatial neighborhood block.

[0029] The occurrence count of existing patterns from spatial neighborhood blocks is accumulated in the histogram.

[0030] Select up to 5 angular patterns and a plane pattern that appear most frequently from HoC, and make a final prediction by mixing the predictions of the selected patterns.

[0031] The calculation of the hybrid weights is similar to that of the DIMD mode, but OBIC does not use the gradient values ​​in the template; instead, it uses the occurrence values. Furthermore, the weights for the planar mode are determined in a similar manner to those of the DIMD mode.

[0032] The present invention discloses a method and apparatus for utilizing inherited information from previously encoded / decoded blocks, wherein the previously encoded / decoded blocks generate TIMD inherited information and / or store TIMD inherited information. [Summary of the Invention] A method and apparatus for video encoding and decoding are disclosed. According to the method, input data associated with the current block is received, wherein the input data includes pixel data to be encoded at the encoder end or encoded data associated with the current block to be decoded at the decoder end. Previously encoded / decoded blocks for the current block are determined, wherein the previously encoded / decoded blocks generated TIMD (Template-based Intra Mode Derivation) information or have stored TIMD information. Inherited TIMD information is derived from the previously encoded / decoded blocks, wherein the inherited TIMD information includes reference line information, wide-angle conditions, or both. Target prediction is generated for the current block using the inherited TIMD information. The current block is encoded or decoded using information including the target prediction.

[0034] In one embodiment, the inheritance information of TIMD is included in a merge candidate list for the current block. In one embodiment, the merge candidate list includes one or more spatially adjacent candidates, one or more non-adjacent candidates, one or more historical candidates, one or more temporal candidates, one or more default candidates, or any combination thereof. In one embodiment, when a new candidate is inserted into the merge candidate list, the merge candidate list is truncated in whole or in part to avoid duplicate inheritance information. In one embodiment, after constructing the merge candidate list, one or more candidates are selected for the current block from the merge candidate list. In one embodiment, the selection of one or more candidates is indicated by an explicit signaling transmission index or determined implicitly.

[0035] In one embodiment, when a TIMD merge mode is applied to the current block, the target prediction for the current block is generated using inherited TIMD information. In another embodiment, whether to apply a TIMD merge mode to the current block depends on the syntax transmitted for the signaling of the current block. In yet another embodiment, when a TIMD merge flag indicates that a TIMD merge mode is applied to the current block, no further signaling is transmitted for the syntax used to determine the prediction information for the current block, and / or the selection of one or more TIMD merge candidates for determining the prediction information for the current block is implicitly determined. [Attached Image Description] Figure 1A An adaptive inter-frame / intra-frame video encoding / decoding system is shown, which includes loop processing.

[0037] Figure 1B It shows Figure 1A The corresponding decoder of the encoder in the code.

[0038] Figure 2A An example of a current block selection template is shown, which includes T rows above the current block and T columns to the left of the current block.

[0039] Figure 2B An example with T=3 is shown, and the HoG (gradient histogram) is calculated for the pixels in the middle row and the pixels in the middle column.

[0040] Figure 2C An example of amplitude (ampl) for angular intra-frame prediction mode is shown.

[0041] Figure 3 The template region for template-based multi-reference line intra-frame prediction is shown. Figure 4 The locations of spatial merging candidates are shown.

[0042] Figure 5 The spatial neighbor blocks used to derive spatial merging candidates are shown.

[0043] Figure 6 An example of several history-based buffer array designs is shown, where each grid represents a CTU.

[0044] Figures 7A-B illustrate how to extract data from a historical table of regions with the same starting geometric position as the current region. Figure 7A An example of constructing a history table for the current region is shown in Figure 7B, which contains a history table of the region with the current region's center geometry.

[0045] Figure 8 A flowchart of an exemplary video codec system for deriving inherited information from previously encoded blocks is shown, according to an embodiment of the present invention.

Detailed Implementation Methods

[0047] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention can be practiced without one or more specific details, or using other methods, components, etc. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the invention. Exemplary embodiments of the invention will be best understood by referring to the accompanying drawings, in which like parts are designated by like numerals throughout the drawings. The following description is merely by way of example and illustrates only embodiments of certain selected apparatuses and methods consistent with the invention.

[0048] II. Proposed Methods This invention proposes an intra-frame merging scheme to improve intra-frame prediction. The concept of the merging scheme is to inherit (or reference) information from previous codec blocks and use this inherited information to predict the current block. The main difference between the proposed intra-frame merging scheme and the conventional intra-frame most probable mode (MPM) is as follows: For intra-frame blocks, the conventional intra-frame most probable mode (MPM) can only inherit intra-frame prediction modes that have been communicated in previous codec blocks, such as intra-frame prediction modes with pointing angle prediction modes from the upper adjacent block and / or the left adjacent block, or mode indices of DC or plane. The proposed intra-frame merging scheme brings a more flexible inheritance process and / or more efficient inheritance information in the following aspects.

[0049] In the first aspect, unlike traditional intra-frame MPM, inherited information is not limited to the intra-frame prediction mode of the signal. In some embodiments entitled "Inherited Information Setting", through the proposed intra-frame merging scheme, inherited information can be any mode information, any sample information, any block information, any model information, and / or any information related to prediction generation.

[0050] Secondly, unlike traditional intra-frame MPMs, the previous codec block containing inherited information is not limited to the left-hand neighboring block or the upper-hand neighboring block. In some embodiments specified in the section entitled "Inheritance Block Setup," the current block can find the previous codec block containing inherited information using several proposed merging methods through the proposed intra-frame merging scheme.

[0051] In the third aspect, unlike the traditional intra-MPM which is only used to improve regular intra-frame luminance prediction, in the section entitled "Target Mode Setting and Shortcut Syntax Setting", for some embodiments, the proposed merging scheme obtains information related to the prediction generation of the current block being merged and decoded by TIMD (the target mode of the proposed intra-frame merging scheme is TIMD).

[0052] When generating a prediction for the current block encoded and decoded by the target mode, one or more successor blocks are selected from a list of merge candidates or from several merge candidates. The prediction for the target mode is then generated using inheritance information from the successor blocks. In some embodiments, the inheritance information for each merge candidate belongs to the same target mode. For example, the inheritance information for each merge candidate is associated with TIMD, and the current block is predicted according to the TIMD process.

[0053] Furthermore, unlike traditional intra-MPMs that rely solely on a limited inheritance method to extract the most probable modes from the angular prediction mode, DC, and plane, the proposed intra-merging scheme incorporates inheritance information settings, inheritance block settings, and / or target mode settings, as well as a shortcut syntax setting. This provides a more efficient and convenient way to obtain information related to prediction generation. In some embodiments, several methods are proposed to define the syntax framework of the proposed intra-merging scheme. Following this framework, the intra-merging scheme can be applied to the current block, and then the prediction for the current block can be generated based on the inheritance information.

[0054] II.1 Inheritance Information Settings The inheritance information setting defines the inheritance information used to generate the current block prediction using the target pattern.

[0055] II.1.1 TIMD Inheritance Information Settings When the intra-frame merging scheme is used for TIMD, that is, when TIMD merging is used for the current block, the inherited information includes the listed items a, b, c, d or any combination thereof.

[0056] One or more TIMD cost values ​​for available TIMD intra-prediction modes (e.g., DC, planar, and / or directional prediction modes); N intra-frame prediction modes suggested by the TIMD cost values ​​(with the minimum N TIMD costs) TIMD weighting information, and / or whether fusion is used Reference line information and / or wide-angle conditions In one embodiment, the directional prediction modes of the intra-frame prediction mode can be located within a predefined directional range. For example, the predefined directional range is 2 to 66, so there are a total of 65 directional prediction modes. As another example, the predefined directional range is 2 to 130, so there are a total of 129 directional prediction modes.

[0057] In another embodiment, the inheritance information comes from a previous codec block specified in the "Inheritance Block Settings" section. The previous codec block generates and / or stores TIMD inheritance information. The TIMD inheritance information stored by the previous codec block can be referenced by subsequent codec blocks (e.g., the current block). For example, for each block containing TIMD information (which can be a TIMD codec block or a non-TIMD codec block), its TIMD information is stored and / or referenced by subsequent codec blocks. For example, for each predefined cell containing TIMD information (which can refer to any predefined region or k×k grid within a TIMD codec block, where k can be 2, 4, 8, 16, or any predefined positive integer), its TIMD information is stored and / or referenced by subsequent codec blocks.

[0058] In one sub-implementation, in response to the inheritance information referring to (a) and / or (b) and / or (c) and / or (d), a TIMD process is performed on the template (neighboring region) of the previously encoded / decoded block to obtain the TIMD cost, thereby deriving the inheritance information of the previously encoded / decoded block. The previously encoded / decoded block may be encoded / decoded using TIMD, TIMD merging, TIMD-related modes, inter-frame, intra-frame, intra-block copy (IBC), any mode type, or any subset / combination of the above.

[0059] In another sub-implementation, in response to the inheritance information referring to (a) and / or (b) and / or (c) and / or (d), a TIMD process is performed on the reconstruction of the previously encoded / decoded block to obtain the cost (distortion between the prediction of the candidate TIMD intra-prediction mode and the reconstruction of the previously encoded / decoded block), thereby deriving the inheritance information of the previously encoded / decoded block. The previous block may be encoded / decoded using TIMD, TIMD merging, TIMD-related modes, inter-frame, intra-frame, intra-block copy (IBC), any mode type, or any subset / combination of the above.

[0060] In another sub-implementation, in response to the inheritance information referring to (a) and / or (b) and / or (c) and / or (d), TIMD-related information for predicting the previous block is obtained to obtain the inheritance information of the previous block. The previous block may be encoded and decoded using TIMD, TIMD merging, or any TIMD-related mode.

[0061] In another embodiment, the inherited information can be used by the current block with some adjustments. Preliminary TIMD information (e.g., weights or TIMD costs) is calculated using the template (neighborhood region) of the current block as regular TIMD. This preliminary TIMD information is used to adjust the inherited TIMD information obtained through Section II.2.1 and / or the re-derived TIMD information obtained through the section entitled "Re-derived Information from Previous Codec Blocks" (an alternative to obtaining inherited TIMD information). The adjustment can be adding the inherited / re-derived TIMD information to the preliminary TIMD information with predefined weights. After adjusting the inherited TIMD information for the current block, the current block can use the adjusted information to obtain one or more intra-prediction modes as regular TIMD.

[0062] In one sub-implementation, the weighting of the initial TIMD information is higher than the weighting of the inherited / re-derived TIMD information. For example, the weighting ratio of the initial TIMD information to the inherited / re-derived TIMD information is 3:1.

[0063] In another sub-implementation, the weighting of the initial TIMD information is lower than the weighting of the inherited / re-derived TIMD information. For example, the weighting ratio of the initial TIMD information to the inherited / re-derived TIMD information is 1:3.

[0064] In another sub-implementation, the weighting varies depending on whether the inherited / re-derived TIMD information is promising. For example, if the reference block used for the inherited / re-derived TIMD information (e.g., a previously encoded / decoded block) is intra-frame encoded / decoded, then the reference block is promising, and the weighting is more biased towards the inherited / re-derived TIMD information.

[0065] In another embodiment, the inherited information can be used by the current block with some adjustments. If multiple inherited TIMD information obtained through Section II.2.1 and / or multiple sets of re-derived TIMD information obtained through Section II.2.2 (an alternative to obtaining inherited TIMD information) are available for the current block, the multiple inherited / re-derived TIMD information are combined with a predefined weight. For example, if the multiple TIMD information comes from the merge candidate list in Section II.2.1, the TIMD information corresponding to the earlier candidates in the merge candidate list (e.g., spatially adjacent candidates) has a higher weight than the other candidates. After adjusting the inherited TIMD information for the current block, the current block can use the adjusted information to obtain one or more intra-prediction modes as regular TIMD.

[0066] In another embodiment, to reduce storage requirements, instead of storing all TIMD information (which can be used for reference), only a subset of the TIMD information is stored. For example, only three or any predefined positive integer of all TIMD information are stored. For instance, the subset is the first three of all TIMD information. The first three may be the three proposed intra-prediction modes.

[0067] II.2 Inheritance Block Settings II.2.1 Inherited information from previous codec blocks Inheritance information is obtained from previous codec blocks. In one embodiment, one or more spatially adjacent candidates and / or non-adjacent candidates, historical candidates, temporal candidates, default candidates, or any subset of the above candidates provide inheritance information from previous codec blocks. In one sub-embodiment, merging candidates references only historical candidates and / or does not use candidates from distant non-adjacent regions.

[0068] In another embodiment, a merge candidate list containing inheritance information is established for the current block. Similar to the regular inter-frame merging mode, the merge candidate list includes spatially adjacent candidates and / or non-adjacent candidates, historical candidates, temporal candidates, default candidates, or any subset of the above. In one sub-embodiment, this merge candidate list is aligned with or a subset of the merge candidate list of the regular inter-frame merging mode. In another sub-embodiment, this merge candidate list references only a history-based buffer array; that is, it includes only historical candidates and / or does not use candidates from distant, non-adjacent regions.

[0069] In one sub-implementation, full or partial pruning is used to avoid duplicate inheritance information in the list. Before adding a candidate to the list, the inheritance information of the candidate to be added, or any subset thereof, is checked against the corresponding inheritance information of the candidates already existing in the list. The full inheritance information of a candidate refers to all stored inheritance information of the inherited block (the previous codec block).

[0070] In another sub-implementation, after establishing a merge candidate list, one or more candidates are selected from the list for use by the current block. The selection depends on an explicit signaling index or implicit selection of one or more (promising) candidates. For example, a successor block encoded / decoded via a target pattern is considered a promising candidate. Alternatively, the top one or more candidates with the lowest cost are selected. For example, an explicit index instructs the reordering of one or more candidates in the list as selected candidates. Cost calculation and / or list reordering may depend on a template matching process that calculates costs based on the distortion between the prediction on the template (using a certain candidate) and the reconstruction on the template. Regarding the reordered list, candidates with the lowest cost are placed at the beginning of the list, or candidates with lower costs are placed further up in the list.

[0071] In another sub-implementation, the merged candidate list has a maximum size threshold. The threshold is fixed in a standard and / or determined based on an explicit syntax at the block, CTU, slice, image, tile, sequence, image, SPS, PPS, or any predetermined unit level. When the threshold is reached (i.e., the list is full), no more candidates can be inserted; otherwise, valid candidates can be inserted into the list. If the list is not full and inheritance cannot find enough candidates, default candidates are inserted to fill the list.

[0072] In another sub-implementation, the size of the merged candidate list is adaptively determined based on the number of valid candidates. When selecting one or more candidates for the current block from the list, the selection is limited to the available candidates in the list. For example, if the selection is implicit, the one or more selected candidates are one or more candidates in the list with lower costs. Cost calculation may depend on a template matching process that calculates the cost based on the distortion between the prediction on the template (using a certain candidate) and the reconstruction on the template.

[0073] II.2.1.1 Spatial Neighbor Candidates and Non-Neighbor Candidates Spatial neighbor candidates are drawn from the neighboring blocks of the current block, where the neighboring blocks can be the same as the five spatial neighbor blocks in the regular inter-frame merge mode or any subset of the neighboring blocks of the current block. Non-neighbor candidates are drawn from the search range around the current block (but not adjacent to it). The search range can be the same as or different from the search range of non-neighbor candidates in the regular inter-frame merge mode.

[0074] II.2.1.2 Historical Candidates Historical candidates are selected from a history-based buffer array. Figure 6 An example of a history-based buffer array design is shown, where each array comprises four CTUs. In the history-based buffer array, inheritance information for each valid previous codec block is stored, where a valid previous codec block refers to any block containing inheritance information. Similar to the history candidates in the merge list of a regular inter-frame merge mode, if the buffer array is full, the first stored information may be removed to include information for the latest valid codec block.

[0075] In one embodiment, the history-based buffer array can be reset at the beginning or end of a predefined cell. Resetting the definition of the history-based buffer array can be by clearing all inherited information stored in the history-based buffer array. The predefined cell can be every M CTUs, every N CTU rows, a slice, a tile, a picture, or any predefined region, where N and M can be any value greater than 0.

[0076] When historical candidates from a history-based buffer array are included in the merge list, the order in which the historical candidates are included can be predefined.

[0077] In one embodiment, the order in which historical candidates are included may be from the beginning / end of the history-based buffer array to the end / beginning of the history-based buffer array.

[0078] In another embodiment, the order in which historical candidates are included may be from a predefined position in the history-based buffer array to the beginning / end of the history-based buffer array.

[0079] In another embodiment, the order in which historical candidates are included can be staggered (e.g., the first included candidate comes from the beginning of the history-based buffer array, the second included candidate comes from the end of the history-based buffer array, and so on).

[0080] In one embodiment, an image, tile, or slice can be divided into several regions, and a history-based buffer array is maintained for each region. During encoding / decoding, the history-based buffer array 0 and an additional history-based buffer array are updated. This additional history-based buffer array can be determined by the current location. For example, if the current CU is located in the second region, the additional history-based buffer array to be updated is history-based buffer array 2.

[0081] In another embodiment, multiple history-based buffer arrays can be used for different update frequencies. For example, the first history-based buffer array updates once per CU, the second history-based buffer array updates once every two CUs, the third history-based buffer array updates once every four CUs, and so on.

[0082] In another embodiment, multiple history-based buffer arrays can be used to store different types of inheritance information. For example, the first history-based buffer array is used to store inheritance information with a single model (i.e., non-fusion case), and the second history-based buffer array is used to store inheritance information with multiple models (i.e., fusion case).

[0083] In another embodiment, multiple history-based buffer arrays can be used for reconstructed samples of different intensities. For example, if the average intensity of reconstructed samples in the current block neighborhood is greater than a predefined threshold, the inherited information will be stored in the first history-based buffer array; otherwise, the inherited information will be stored in the second history-based buffer array.

[0084] When incorporating historical candidates from multiple history-based buffer arrays into the merge list, the order in which they are included can be predefined.

[0085] In one embodiment, the order in which historical candidates are included can be: from the beginning / end of a history-based buffer array to the end / beginning of that history-based buffer array, and then the next history-based buffer array is included in the same order or in the reverse order.

[0086] In another embodiment, the order of incorporating historical candidates can be: from a predefined position of a history-based buffer array to the end / start of the history-based buffer array, and then incorporating the next history-based buffer array in the same order or in the reverse order.

[0087] In another embodiment, the order of incorporating historical candidates can be: in a history-based buffer array, in an interleaved manner (e.g., the first candidate to be incorporated comes from the beginning of the history-based buffer array, the second candidate to be incorporated comes from the end of the history-based buffer array, and so on), and then in the same order or in the reverse order, in the next history-based buffer array.

[0088] In another embodiment, the order in which history candidates are included can be: from the beginning / end of each history-based buffer array to the end / beginning of each history-based buffer array.

[0089] In another embodiment, the order in which history candidates are included can be: from a predefined position in each history-based buffer array to the end / start of each history-based buffer array.

[0090] In another embodiment, the order in which historical candidates are included can be: in an interleaved manner in each history-based buffer array (e.g., the first candidate included comes from the beginning of the history-based buffer array, the second candidate included comes from the end of the history-based buffer array, and so on).

[0091] In one embodiment, multiple history-based buffer arrays exist, but not all of them will be used to build the merge list. Only history-based buffer arrays whose regions are close to the region where the current block is located can be used to build the merge list.

[0092] In one embodiment, if historical candidates are included in the merge list, the range of non-adjacent candidates can be narrowed by using smaller distances between the locations of each non-adjacent candidate.

[0093] In another embodiment, if historical candidates are included in the merge list, the number of non-adjacent candidates can be reduced by measuring the distance from the top left position of the current block to the candidate position and excluding candidates whose distance is greater than a predefined threshold.

[0094] In another embodiment, if historical candidates are included in the merge list, the number of non-adjacent candidates can be reduced by skipping candidates that are not located in the same region.

[0095] In another embodiment, if historical candidates are included in the merge list, the number of non-adjacent candidates can be reduced by skipping candidates that are not located in the same or neighboring regions. The range of neighboring regions is predefined and can be M×N regions (M by N regions), where M and N can be any value greater than 0.

[0096] In another embodiment, if historical candidates are included in the merge list, the search scope for selecting non-adjacent candidates can be narrowed by skipping the second search mode.

[0097] In one embodiment, an image can be divided into multiple regions, and at least one history table is maintained in each region. For a region of the current image, it can use or combine history tables from one or more regions of previously encoded images as the initial history table. For example, if an image is divided into N regions, it can implicitly or explicitly select a history table from one of the N regions of previously encoded images as the initial history table. The index of one of the N regions can be transmitted by signaling or can be implicitly derived from the corresponding region in the previously encoded images. Figure 7A and Figure 7B As shown, the current image 720 is a P / B codec image, while the previous image 710 was an intra-frame codec image. Each image is divided into four regions as shown by four rectangular boxes. According to an embodiment of the present invention, the corresponding region in the previously codec image can be... Figure 7A The region 712 shown has the same initial geometric position as the current region 722, or is... Figure 7B The area shown is region 712, which contains the central geometric location of the current region 722. For example, it can combine more than one history table from previously encoded / decoded regions / images to construct a history table for the current region.

[0098] II.2.1.3 Time Candidates Timing candidates are obtained from inherited information stored in one or more previously encoded / decoded images. Timing candidates can be obtained when the current slice / image is a non-intra-frame slice / image.

[0099] In one embodiment, the time candidate can be derived from certain predefined locations of previously encoded / decoded slices / images. The block.

[0100] In one sub-implementation, location Within the corresponding region of the current encoding / decoding block, i.e. and Predefined positions can be .

[0101] In one sub-implementation, a predefined location Located outside the corresponding region of the current encoding / decoding block, i.e. and / or The time candidate can be a previously encoded / decoded slice / image. .

[0102] In one sub-implementation, the predefined position can be determined based on the position, width, and height of the current block.

[0103] In one embodiment, the previously encoded / decoded image is located among the images in the reference list.

[0104] In one embodiment, the previously encoded / decoded image is the same as the collocated image in a regular intra-frame merging mode.

[0105] In one embodiment, the previously encoded / decoded image is determined based on motion vectors of neighboring blocks. This image is the one indicated by the motion vectors.

[0106] In one embodiment, the final predefined position is determined based on the motion vectors of neighboring blocks. Horizontal and vertical displacements (Δx, Δy) are determined based on the motion vectors. Assume a set of predefined positions is (x1, y1), (x2, y2), ..., (xN, yN). The final predefined position could be (x1+Δx, y1+Δy), (x2+Δx, y2+Δy), ..., (xN+Δx, yN+Δy).

[0107] In one embodiment, the motion vector can be the L0 or L1 motion vector of a neighboring block.

[0108] In one embodiment, the horizontal and vertical displacements (Δx, Δy) are derived from selected neighboring motion vectors. For example, if the reference image of the selected neighboring motion vector and the selected / predefined previously encoded image are the same image, (Δx, Δy) is equal to the neighboring motion vector or the rounded neighboring motion vector. Another example is if the reference image of the selected motion vector and the selected / predefined previously encoded image are not the same image, (Δx, Δy) can be (mv_x * (td / tb), mv_y * (td / tb)) or the rounded (mv_x * (td / tb), mv_y * (td / tb)), where (mv_x, mv_y) is the selected motion vector, tb is the POC distance between the current image and the reference image of the selected motion vector, and td is the POC distance between the current image and the selected / predefined previously encoded image.

[0109] In one embodiment, the temporal candidate is derived using reconstructed samples of blocks located at predefined positions within a selected / predefined previously encoded image. The block size can be determined by the size of the current block. For example, the size can be the same as the current block size. The block size can also be fixed. For example, the size can always be M x N, where M and N are positive integers.

[0110] II.2.1.4 Default Candidate A default candidate is a candidate that contains default information and / or the default information is derived from candidates already included in the merged candidate list. In one embodiment, the default mode of the default candidate can be derived using the MIMD scheme, the OBIC scheme, the EIP derived mode scheme, any existing or mentioned intra-frame scheme (e.g., TIMD, DIMD, EIP and / or MRL), any variant of any existing or mentioned intra-frame scheme, or any combination thereof.

[0111] II.2.2 Re-exporting information from previous codec blocks If the current block decides to reference inherited information from a previous codec block (e.g., to select a candidate from a merge candidate list), the export procedure is applied to the reference block (i.e., the previous codec block) to re-extract the required inherited information from that reference block. Through this re-extraction, the inherited information from the previous codec block does not need to be stored in the previous codec block. This re-extraction can replace, or conditionally replace, the method of storing inherited information.

[0112] II.2.3 Constraints Inherited / Re-derived (Validity Check) With constraints in place, when attempting to obtain inherited information from a previous codec block, the inherited information can only be inherited / re-exported to the current block if the constraints are met.

[0113] In one embodiment, only inherited information from intra-frame codecs, target mode codecs, target merge mode codecs, target mode-related codec blocks, or combinations thereof, is available. Only inherited information from blocks located in predefined regions is available. For example, a predefined region is determined by the current block's block position, block width, block height, and / or block area.

[0114] II.3 Target Mode Settings and Shortcut Syntax Settings In one embodiment, when the proposed intra-frame combining scheme is used for the current block, the predictor sub-DIMD of the current block is used to select the transform set and / or transpose flags of the primary transform and / or secondary transform.

[0115] II.3.1 TIMD Target Mode Settings (TIMD Merge Mode) The target mode setting for TIMD specifies a TIMD merging mode. When this TIMD merging mode is used for the current block, the current block generates its prediction using inherited TIMD information. This inherited information can be adjusted or unadjusted. Inherited information can refer to: N intra-prediction modes (e.g., IPM1 and IPM2) for available TIMD intra-prediction modes, TIMD weighting information (e.g., W1 and W2), information on whether merging is performed, reference line information, wide-angle conditions, or any combination thereof. The current block obtains its own prediction information from this inherited information. Depending on the inherited block setting, inherited information from one or more predefined candidates (e.g., each candidate provides {IPM1, IPM2, W1, W2}) is used to determine the prediction information needed to generate the current block's prediction. For example, the merge candidate list is constructed and rearranged based on the cost during the template matching process. Template size and template cost calculations can be unified with regular TIMD. After rearranging the TIMD merge candidates, the top two or any predefined number of candidates are retained from the list for signaling transmission. Then, in accordance with regular TIMD, the prediction hypotheses from each inherited intra-prediction mode are combined through a mixing process to form the final prediction for the current block.

[0116] In one embodiment, the cost for a candidate during template matching is calculated based on the candidate's template prediction and template reconstruction, wherein the candidate's template prediction is a weighted average of two template predictions from IPM1 and IPM2 using two weights, W1 and W2.

[0117] In another embodiment, the cost for a candidate during template matching is calculated based on: (1) the distortion between the template prediction and template reconstruction obtained from IPM1 for that candidate, and (2) the distortion between the template prediction and template reconstruction obtained from IPM2 for that candidate. A specific example of the cost for that candidate could be: ((distortion of IPM1) × W1 + (distortion of IPM2) × W2).

[0118] In another embodiment, based on the inherited block settings, only one candidate, such as the first available candidate, is used to determine the prediction information for the current block.

[0119] In another embodiment, one or more candidates are used to determine the prediction information for the current block. For example, all available candidates are used to determine the prediction information for the current block. For example, a predefined maximum number is set. During the checking of available candidates, when the number of available candidates checked reaches the predefined maximum number, the checked available candidates are used to determine the prediction information for the current block. After checking all available candidates (the number of available candidates checked may or may not have reached the predefined maximum number), the checked available candidates are used to determine the prediction information for the current block.

[0120] In another embodiment, whether the TIMD merge mode is applied to the current block depends on the syntax of the current block. For example, a flag (such as the TIMD merge flag) is signaled to indicate whether the TIMD merge mode is applied to the current block.

[0121] In one sub-implementation, when the TIMD merge flag indicates that the TIMD merge mode should be applied to the current block, the candidate selection of no further communication syntax and / or decision of the prediction information for the current block is implicit.

[0122] In another sub-implementation, when the TIMD merge flag indicates that the TIMD merge mode should be applied to the current block, further signaling of one or more syntax elements and / or determining candidate options for the prediction information of the current block is explicit.

[0123] In another sub-implementation, when the TIMD merge flag indicates that the TIMD merge mode should be applied to the current block, one or more syntax elements are further messaged and / or an indicated TIMD merge mode is selected from several TIMD merge mode variants based on the messaged syntax elements.

[0124] In another sub-implementation, the TIMD merge mode is an optional mode of regular TIMD, and the TIMD merge flag is signaled after the enable flag of regular TIMD. For example, the TIMD merge flag (as a shortcut to TIMD) is signaled immediately after the enable flag of regular TIMD, or before all or a subset of the submodes of regular TIMD. The TIMD merge flag is further signaled only if the enable flag of regular TIMD indicates that regular TIMD should be applied to the current block; otherwise, the TIMD merge flag is inferred to be disabled.

[0125] In this invention, the term "block" may refer to TU / TB, CU / CB, PU / PB, a predefined region, or CTU / CTB.

[0126] Any combination of the methods proposed in this invention can be applied.

[0127] The method proposed in this invention can be enabled and / or disabled based on implicit rules (e.g., block width, height, or area) or explicit rules (e.g., block, slice, image, SPS, or PPS level syntax). For example, the proposed method is applied when the block area is less than / greater than a threshold.

[0128] Any of the methods proposed above for deriving TIMD inheritance information from previous codec blocks can be implemented in the encoder and / or decoder. For example, any of the proposed methods can be implemented in the inter-frame / intra-frame / IBC / prediction / transform module of the encoder, and / or in the inter-frame / intra-frame / IBC / prediction / transform module of the decoder. Alternatively, any of the proposed methods can be implemented as circuitry connected to the inter-frame / intra-frame / IBC / prediction / transform module of the encoder and / or the inter-frame / intra-frame / IBC / prediction / transform module of the decoder to provide the information required by the inter-frame / intra-frame / IBC / prediction / transform module.

[0129] Reference Figure 1A and Figure 1B The exemplary encoder and decoder in the example, any proposed candidate derivation method can be implemented in the intra-frame encoding / decoding module in the decoder (e.g., Figure 1B Intra-frame prediction 150 in the encoder or intra-frame codec module in the encoder (e.g., Figure 1A Intra-frame prediction (110) is implemented in the decoder. Any of the proposed methods can also be implemented as circuitry connected to an intra-frame codec module in the decoder or encoder. However, the decoder or encoder can also use additional processing units to implement the required processing. Although the intra-frame prediction unit (e.g., Figure 1A Unit 110 and Figure 1BUnit 150 in the diagram is shown as an independent processing unit, which may correspond to executable software or firmware code stored on a medium, such as a hard disk or flash memory, for a central processing unit (CPU) or a programmable device (e.g., a digital signal processor (DSP) or a field-programmable gate array (FPGA)).

[0130] Figure 8 A flowchart illustrating an exemplary video codec system for deriving inherited information from previously encoded blocks in a video codec system according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented at the encoder end by program code executed by one or more processors (e.g., one or more CPUs). The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors configured to perform the steps in the flowchart. According to the method, in step 810, input data associated with the current block is received, wherein the input data includes pixel data to be encoded at the encoder end or encoded data associated with the current block to be decoded at the decoder end. In step 820, a previously encoded block for the current block is determined, wherein the previously encoded block generates TIMD (Template-based Intra Mode Derivation) information or has stored TIMD information. In step 830, inherited information of the TIMD is derived from the previously encoded block, wherein the inherited information of the TIMD includes reference line information, wide-angle conditions, or both. In step 840, the inherited information of the TIMD is used to generate target prediction for the current block. In step 850, the current block is encoded or decoded using information including the target prediction.

[0131] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. Those skilled in the art can practice the invention by modifying each step, rearranging the steps, splitting the steps, or merging the steps without departing from the spirit of the invention. Specific syntax and semantics are used in this disclosure to illustrate examples of implementing embodiments of the invention. Those skilled in the art can practice the invention by using equivalent syntactic and semantic substitutions without departing from the spirit of the invention.

[0132] The foregoing description is intended to enable those skilled in the art to practice the invention in the context of specific applications and their requirements. Various modifications to the described embodiments will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but should be given the broadest scope in accordance with the principles and novel features disclosed herein. In the foregoing detailed description, various specific details have been shown to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced.

[0133] The embodiments of the present invention described above can be implemented in various hardware, software code, or combinations thereof. For example, embodiments of the invention may be one or more electronic circuits integrated into a video compression chip, or program code integrated into video compression software, to perform the processes described herein. Embodiments of the invention may also be program code executed on a digital signal processor (DSP) to perform the processes described herein. The invention may also relate to multiple functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). These processors can be configured to execute machine-readable software code or firmware code to define specific methods embodied by the invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles, and languages ​​for performing tasks according to the invention, as well as other methods of configuring the code, do not depart from the spirit and scope of the invention.

[0134] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The examples described are for illustrative purposes only and not for limitation. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All modifications within the meaning and equivalence of the claims should be included within its scope.

Claims

1. A video encoding / decoding method, comprising: Receive input data related to the current block, wherein the input data includes pixel data to be encoded at the encoder end or encoded data related to the current block to be decoded at the decoder end; Determine the previously encoded / decoded block used for the current block, wherein the previously encoded / decoded block generates TIMD (Template-based Intra Mode Derivation) information or has stored TIMD information; The inherited information of the TIMD is derived from the previously encoded / decoded block, wherein the inherited information of the TIMD includes reference line information, wide-angle conditions, or both. Use the inherited information from the TIMD to generate the target prediction for the current block; as well as The current block is encoded or decoded using encoding / decoding information that includes the target prediction.

2. The method of claim 1, wherein the inheritance information of the TIMD is included in the merge candidate list for the current block.

3. The method of claim 2, wherein the merged candidate list includes one or more spatially adjacent candidates, one or more non-adjacent candidates, one or more historical candidates, one or more time candidates, one or more default candidates, or any combination thereof.

4. The method of claim 2, wherein when a new candidate is inserted into the merged candidate list, the merged candidate list is pruned in whole or in part to avoid duplicate inheritance information.

5. The method of claim 2, wherein after constructing the merge candidate list, one or more candidates are selected from the merge candidate list for the current block.

6. The method of claim 5, wherein the selection of the one or more candidates is indicated by an explicit signaling transmission index or determined implicitly.

7. The method of claim 1, wherein when the TIMD merge mode is used on the current block, the current block uses the inheritance information of the TIMD to generate the target prediction of the current block.

8. The method of claim 7, wherein whether the TIMD merge mode is applied to the current block depends on the syntax of the signaling for the current block.

9. The method of claim 7, wherein when the TIMD merge flag indicates that the TIMD merge mode is applied to the current block, no further signaling transmission is required for the syntax of the prediction information for the current block and / or the selection of one or more TIMD merge candidates for determining the prediction information for the current block is implicitly determined.

10. A video encoding / decoding apparatus, comprising one or more electronic devices or processors, configured to: Receive input data related to the current block, wherein the input data includes pixel data to be encoded at the encoder end or encoded data related to the current block to be decoded at the decoder end; Determine the previously encoded / decoded block used for the current block, wherein the previously encoded / decoded block generates TIMD (Template-based Intra Mode Derivation) information or has stored TIMD information; The inherited information of the TIMD is derived from the previously encoded / decoded block, wherein the inherited information of the TIMD includes reference line information, wide-angle conditions, or both. Use the inherited information from the TIMD to generate the target prediction for the current block; as well as The current block is encoded or decoded using encoding / decoding information that includes the target prediction.

Images