Filter derivation method and apparatus for intra-frame prediction based on filters in video encoding and decoding systems

Abstract

A method and apparatus for video encoding and decoding using a filter-based intra-prediction mode are disclosed. According to the method, a reference block for the current block is determined. A residual of the reference block is determined, corresponding to the difference between filtered prediction samples and reconstructed samples of the reference block, wherein the filtered prediction samples of the reference block are generated by applying a filter-based target intra-prediction mode to the reference block. Based on the residual of the reference block, a filter-based target intra-prediction associated with the filter-based target intra-prediction mode is modified into a modified filter-based prediction for the current block. A modified current filtered prediction is generated by applying the modified filter-based prediction for the current block to the current block. According to another method, a prediction for the current block is generated using stored filter parameters derived by matching the filtering results with reconstructed samples of a previous reconstructed block.

Description

[0001] Cross-referencing This invention is a non-provisional application and claims priority to U.S. Provisional Patent Application No. 63 / 621,640, filed January 17, 2024. The entire contents of that U.S. Provisional Patent Application are hereby incorporated. [Technical Field] This invention relates to video encoding and decoding systems. Specifically, this invention relates to a scheme for improving filter-based intra-frame prediction performance by using the residuals of reference blocks or by using filter parameters derived and stored for previously reconstructed blocks. [Background Technology] Universal Video Coding (VVC) is the latest international video coding standard (MPEG) developed by the Joint Video Experts Group (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group. This standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology - Coded representation of immersive media - Part 3: Versatile video coding, published in February 2021. VVC was developed by adding more coding and decoding tools to improve coding and decoding efficiency and handle various types of video sources, including 3D video signals, based on its predecessor, HEVC (High Efficiency Video Coding).

[0004] Figure 1AAn exemplary adaptive inter / intra video codec system incorporating loop processing is described. For intra-frame prediction, prediction data is derived from previously encoded video data in the current frame (hereinafter referred to as a frame). For inter-frame prediction 112, motion estimation (ME) is performed on the encoder side, and motion compensation (MC) is performed based on the result of ME to provide prediction data derived from other frames and motion data. Switch 114 selects either intra-frame prediction 110 or inter-frame prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The residual from transform and quantization is then encoded by entropy encoder 122 to be included in the video bitstream corresponding to compressed video data. Then, the bitstream associated with the transform coefficients is packaged together with side information (e.g., motion and decoding modes associated with intra-frame and inter-frame prediction) and other information (e.g., parameters associated with the loop filter applied to the underlying image area). The side information associated with intra-frame prediction 110, inter-frame prediction 112, and loop filter 130 is provided to the entropy encoder 122, such as... Figure 1A As shown. When using inter-frame prediction mode, one or more reference images must also be reconstructed at the encoder end. Therefore, the transformed and quantized residuals are processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residuals. Then, at reconstruction (REC) 128, the residuals are added back to the prediction data 136 to reconstruct the video data. The reconstructed video data can be stored in the reference image buffer 134 and used to predict other frames.

[0005] like Figure 1AAs shown, the input video data undergoes a series of processing steps in the encoding system. Due to these processing steps, the reconstructed video data from REC 128 may suffer various forms of degradation. Therefore, loop filters 130 are frequently applied to the reconstructed video data before it is stored in the reference image buffer 134 to improve video quality. For example, deblocking filters (DF), Sample Adaptive Offset (SAO), and Adaptive Loop Filters (ALF) may be used. It may be necessary to incorporate loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, loop filter information is also provided to the entropy encoder 122 for incorporation into the bitstream. Figure 1A In the process, the loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in the reference image buffer 134. Figure 1A The system described herein is intended to illustrate an exemplary architecture of a typical video encoder. It may correspond to a High Efficiency Video Codec (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0006] like Figure 1B As shown, apart from transform 118 and quantization 120, the decoder can use similar or identical function blocks to the encoder, since the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and the necessary encoding / decoding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not require mode search. Instead, the decoder only needs to generate intra-frame predictions based on the intra-frame prediction information received from entropy decoder 140. Furthermore, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from entropy decoder 140 without motion estimation.

[0007] Similar to HEVC, VVC partitions the input image into non-overlapping square block regions called CTUs (Code-Decoder Tree Units). Each CTU can be further divided into one or more smaller Code-Decoder Units (CUs). The resulting CU partitions can be squares or rectangles. Furthermore, VVC divides the CTUs into Prediction Units (PUs) as units for applying prediction processing, such as inter-frame prediction, intra-frame prediction, etc.

[0008] CTU partitioned using a tree structure In VVC, a quadtree partitioning structure using nested multi-type trees with binary and ternary partitions replaces the concept of multiple partitioning unit types. This eliminates the separation between CU, PU, ​​and TU concepts, except for CUs larger than the maximum transform length, and supports more flexible CU partition shapes. In the coding tree structure, CUs can be square or rectangular. Coding tree units (CTUs) are first partitioned using a quadtree structure. Then, the leaf nodes of the quadtree can be further partitioned using multi-type tree structures. For example... Figure 2 As shown, there are four partition types in the multi-type tree structure: vertical binary partition (SPLIT_BT_VER 210), horizontal binary partition (SPLIT_BT_HOR 220), vertical ternary partition (SPLIT_TT_VER 230), and horizontal ternary partition (SPLIT_TT_HOR 240). The leaf nodes of the multi-type tree are called codec units (CUs), and unless the CU is excessively large for the maximum transform length, this partition will be used for prediction and transform processing without further partitioning. This means that in most cases, in the coded block structure of this quadtree nested multi-type tree, the CU, PU, ​​and TU have the same block size. Exceptions occur when the maximum supported transform length is less than the width or height of the color component of the CU.

[0009] Intra-mode codec with 67 intra-prediction modes To capture arbitrary edge directions presented in natural video, the number of directional intra modes in VVC has been expanded from 33 used in HEVC to 65.

[0010] Decoder Side Intra Mode Derivation (DIMD) When DIMD is applied, two intra-frame modes are derived from the reconstructed neighboring samples (templates), and these two predictors are combined with the planar mode predictors using weights derived from the gradient.

[0011] Texture gradient analysis is performed at both the encoder and decoder ends. This process begins with an empty Histogram of Gradients (HoG) with 65 entries, corresponding to 65 angular patterns. The amplitudes of these entries are determined during texture gradient analysis.

[0012] For more details, please refer to JVET-O0449 “Non-CE3: Decoder-side Intra Mode Derivation (DIMD) with prediction fusion using Planar” and / or “Algorithm description of Enhanced Compression Model” (e.g., JVET-AG2025) and / or JVET-AB0116.

[0013] Template-based Intra Mode Derivation (TIMD) Template-based intra-frame mode derivation (TIMD) mode uses neighboring templates at the encoder and decoder to implicitly derive the intra-frame prediction mode of the CU, instead of communicating the intra-frame prediction mode to the decoder.

[0014] For more details, please refer to the "Algorithm description of Enhanced Compression Model" published by JVET-AG2025 and / or JVET-AG0092.

[0015] Extrapolation Filter-Based Intra Prediction (EIP) mode (from JVET-AF0080) In JVET-AF0080 (Luhang Xu et al., “EE2-2.7: An extrapolation filter-based intra prediction mode”, ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 Joint Video Experts Team (JVET), 32nd meeting, Hannover, DE, October 13-20, 2023, document: JVET-AF0080), intra prediction based on an extrapolation filter was disclosed, in which EIP prediction was performed in three steps.

[0016] A. Obtaining the EIP filter exist Figure 3 The image shows three EIP filter shapes, which correspond to square 310, horizontal bar 320, and vertical bar 330, respectively.

[0017] There are two methods to obtain the filter coefficients of the current CU. One: The coefficients can be derived from neighboring reconstructed pixels. Two: The coefficients can be inherited from previously decoded blocks.

[0018] B. Deriving the EIP coefficients The decoder decodes the relevant syntax elements to determine the type of the selected reconstruction region and the filter shape of the current block. The selected filter moves horizontally or vertically within the selected reconstruction region with a stride of one pixel to construct the autocorrelation matrix and cross-correlation vector. The process of calculating the coefficients from the autocorrelation matrix and cross-correlation vector is the same as that in the convolutional cross-component model (CCCM).

[0019] The three types of reconstructed regions are shown in Figure 4, where the three reconstructed regions correspond to the upper left (Left-Above) region. Figure 4A Above area Figure 4B ) and the left (Left) area ( Figure 4C The size of the reconstructed region depends on min(blockWidth, blockHeight) and the selected filter shape. For example, when the current block is an 8x16 block and the selected filter shape is 4x4, the aboveSize of the reconstructed region is equal to min(8, 16) + 4 - 1 = 11, and the leftSize of the reconstructed region is equal to min(8, 16) + 4 - 1 = 11.

[0020] C. Inheritance of EIP Filters The filter shape and filter coefficients can be inherited from previously decoded blocks with EIP or EIP merging modes. The decoder decodes an EIP merging flag to determine whether to use the proposed merging mode when the current block uses an EIP mode. When the EIP merging flag is true, the merging index is further decoded. The EIP merging list includes spatially adjacent and non-adjacent candidates, temporal candidates, and historical candidates. The constructed EIP merging list can contain up to 12 candidates, and through a reordering process based on the SAD cost measured on an L-shaped template with a column width and row height of 1, the list can be reduced to a maximum of 6 candidates. In SAD computation, predictions for the template region are generated only from the reconstructed (neighboring and template) samples, allowing EIP filters to be applied in parallel rather than sequentially.

[0021] D. Prediction of the current block EIP mode generates predicted values ​​for the current block by predicting diagonally from the top left to the bottom right position, such as... Figure 5 As shown.

[0022] The predicted values ​​are calculated as follows: , in It is the predicted value at (x, y) in the current block. It is the i-th of the selected EIP filter Coefficients, with indices ranging from 0 to 14. It is the reconstructed or predicted value used for current location prediction. and These are the position offsets from the current position along the x and y directions, respectively.

[0023] Inter-frame prediction For each inter-frame prediction CU, motion parameters include motion vectors, reference image indices, reference image list usage indices, and additional information required by the new coding features of VVC when generating inter-frame prediction samples. Motion parameters can be signaled explicitly or implicitly. When a CU is encoded / decoded in skip mode, it is associated with a PU and has no significant residual coefficients, no encoded motion vector differences, or reference image indices. A merge mode is specified, allowing motion parameters for the current CU to be obtained from neighboring CUs, including spatial and temporal candidates, as well as additional scheduling introduced in VVC. The merge mode can be applied to any inter-frame prediction CU, not just skip mode. An alternative to the merge mode is explicit transmission of motion parameters, where for each CU, the motion vectors, reference image indices corresponding to each reference image list, reference image list usage flags, and other necessary information are explicitly signaled.

[0024] In addition to HEVC's inter-frame coding features, VVC also includes a variety of new and improved inter-frame prediction coding and decoding tools, listed below: – Extended merge prediction – Merge pattern with MVD (MMVD) Symmetric MVD (SMVD) signaling – Affine motion compensation prediction – Sub-block-based temporal motion vector prediction (SbTMVP) – Adaptive Motion Vector Resolution (AMVR) – Sports field storage: 1 / 16 luminance sample MV storage and 8×8 sports field compression – Bidirectional prediction (BCW) with CU-level weights – Bidirectional optical flow (BDOF) – Decoder-side motion vector refinement (DMVR) – Geometric Partitioning (GPM) – Combined Inter-Frame and Intra-Frame Prediction (CIIP) For more details, please refer to JVET-T2002 “Algorithm description for Versatile VideoCoding and Test Model 11 (VTM 11)”.

[0025] The present invention discloses a method and apparatus for improving filter-based prediction performance by using the residuals of a reference block or by using filter parameters derived and stored for a previously reconstructed block. [Summary of the Invention] A method and apparatus for video encoding and decoding are disclosed. According to the method, input data related to a current block is received, wherein the input data includes pixel data to be encoded at an encoder end, or encoded data related to the current block to be decoded at a decoder end. A reference block for the current block is determined. A residual of the reference block is determined, corresponding to the difference between a filtered prediction sample of the reference block and a reconstructed sample of the reference block, wherein the filtered prediction sample of the reference block is generated by applying a target filter-based intra-prediction mode to the reference block. Based on the residual of the reference block, a target filter-based intra-prediction associated with the target filter-based intra-prediction mode is modified into a modified filter-based prediction for the current block. A modified current filtered prediction is generated by applying the modified filter-based prediction for the current block to the current block.

[0027] In one embodiment, the target filter-based intra-prediction mode corresponds to EIP (Extrapolation Filter-Based Intra-Prediction). In one embodiment, the modified filter-based prediction for the current block is determined by subtracting or adding the residual to the current prediction. In one embodiment, the modified current filtered prediction is combined with a filtered prediction sample from a reference block to generate the final prediction for the current block. In one embodiment, information associated with the target filter-based intra-prediction mode includes filter shape and filter parameters.

[0028] According to another method, stored filter parameters associated with a target filter for a filter-based prediction mode are determined, wherein the stored filter parameters are derived by matching the filtering result with reconstructed samples of a previous reconstructed block, and the filtering result is generated by applying the target filter to the prediction of the previous block. The target filter using the stored filter parameters is applied to the initial predictor of the current block to generate the current predictor of the current block. The current block is then encoded or decoded using encoding information including the current predictor.

[0029] In one embodiment, the initial predictor is generated by intra prediction or non-intra prediction. In one embodiment, the initial predictor corresponds to a reconstructed sample or a predicted sample before the current to-be-filtered predicted sample. In one embodiment, the previous reconstructed block corresponds to one or more templates of one or more previous blocks. In one embodiment, the previous reconstructed block corresponds to an adjacent reference region or a non-adjacent region. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1A Illustrate an exemplary adaptive inter / intra video coding system, which includes loop processing.

[0031] Figure 1B Illustrate Figure 1A the corresponding decoder of the encoder.

[0032] Figure 2 Illustrate four splitting types in a multi-type tree structure, including vertical binary splitting (SPLIT_BT_VER), horizontal binary splitting (SPLIT_BT_HOR), vertical ternary splitting (SPLIT_TT_VER), and horizontal ternary splitting (SPLIT_TT_HOR).

[0033] Figure 3 Illustrate three filter shapes, each having fifteen inputs and generating an output for an EIP process.

[0034] Figures 4A-4C Illustrate three Figure 4A : upper left region, Figure 4B : upper region, and Figure 4C : left region) of the reconstructed regions for deriving the filter coefficients of the EIP.

[0035] Figure 5 Illustrate an example of a scan order for generating predictions at different positions of a current block in diagonal order.

[0036] Figure 6 Illustrate an example of a filter shape using a pattern around (excluding) the position (x, y) of the to-be-predicted sample, where the pattern is an MxN region.

[0037] Figure 7 Illustrate an example of an MxN square region, excluding regions where the horizontal coordinate > x and the vertical coordinate > y.

[0038] Figure 8 Illustrate an example of a non-square region (M < N), excluding regions where the horizontal coordinate > x and the vertical coordinate > y.

[0039] Figure 9 Illustrate an example of a non-square region (M > N), excluding regions where the horizontal coordinate > x and the vertical coordinate > y.

[0040] Figure 10 This example illustrates the template (i.e., the neighboring region) of the current block used to derive filter parameters, where the template refers to the top template, the left template, and the top-left template.

[0041] Figure 11 This illustrates an example of a sample to be predicted near the top boundary of the current block.

[0042] Figure 12 This provides an example of a sample to be predicted that is inside the current block and / or far from the top and left boundaries.

[0043] Figure 13 shows an example in which the current block is filtered and predicted using an intra-frame prediction mode based on the target filter, and the filtered prediction reference (filter_pred_ref) is combined with the combined filtered prediction to generate the final prediction for the current block.

[0044] Figure 14 shows an example of filter derivation based on a reconstruction block, which uses the residual between the filtered predicted samples and the reconstructed samples of the reconstruction block to modify the filter parameters of the current block.

[0045] Figure 15 shows an example of deriving filter parameters using non-adjacent reference regions.

[0046] Figure 16 illustrates a flowchart of an exemplary video encoding and decoding system according to an embodiment of the present invention, which uses the residual of a reference block to modify the filter prediction of the current block.

[0047] Figure 17 shows a flowchart of an exemplary video encoding and decoding system according to an embodiment of the present invention, which uses stored filter parameters obtained by matching the filtering results with reconstructed samples of previous reconstructed blocks.

Detailed Implementation Methods

[0049] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the invention can be practiced without using one or more specific details, or using other methods, components, etc. In other instances, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the invention. Embodiments of the invention can be best understood by referring to the drawings, where identical parts are designated by the same numbers throughout. The following description is merely illustrative and only illustrates embodiments of selected apparatus and methods consistent with the invention declared herein.

[0050] The proposed method In this invention, filter-based target intra-frame prediction modes and / or filter-based target novel prediction modes can be used to generate predictions for intra-frame blocks or non-intra-frame blocks, where non-intra-frame blocks refer to inter-frame modes and / or block vector modes. For example, a block vector mode refers to intra-frame block duplication or intra-frame TMP, which searches within a predefined region of the current image and finds a reference block (whose template matches the template of the current block) within that predefined region to predict the current block. To obtain filters for filter-based target intra-frame prediction modes or filter-based target novel prediction modes, one approach is from a derivation path and / or another is from an inheritance path.

[0051] When a filter-based target intra-prediction mode or a filter-based target novel prediction mode is used on the current block, obtaining an efficient filter can improve encoding and decoding performance. Therefore, several embodiments are proposed to obtain an efficient filter for the current block. In one embodiment, the filter-based target intra-prediction mode is EIP. In another embodiment, the filter-based target intra-prediction mode is not limited to the EIP shown in the background section, and / or can be any variant of the filter-based intra-prediction mode. In Section (1), several variants of the filter-based intra-prediction mode are proposed as filter-based target intra-prediction modes. In Section (2), a filter-based target novel prediction mode is described. In Section (3), a method for obtaining an efficient filter is proposed. In Section (4), several methods are proposed to define the storage rules for the filter-based target intra-prediction mode or the filter-based target novel prediction mode.

[0052] (1) Filter-based intra-frame prediction mode The filter (or model) for the target intra-frame prediction mode based on the filter includes the determination of the filter shape and / or filter parameters. After determining the filter shape in one or more candidate filter shapes and / or the filter parameters in the determination of one or more candidate filter parameters, the filter for the current block is obtained. For each sample to be predicted in the current block, the filter is applied. In one embodiment, the filter application follows a predefined order, such as horizontal scan, vertical scan, diagonal scan, and / or any scan order. For example, the predefined order is diagonal scan. That is, the predicted sample located at the first position in the diagonal scan of the current block is output first. Then, the predicted samples located at the second, third, ..., nth positions in the diagonal scan of the current block are output sequentially, where n corresponds to the block size. When applying the filter, the filter input is combined (or convolved) with the filter parameters to obtain the filter output. In one embodiment, the input samples refer to the reconstructed and / or filtered prediction and / or predicted samples associated with the current block and / or the reconstructed and / or filtered prediction and / or predicted samples associated with a reference region reconstructed before or adjacent to the current block. Finally, after applying the filter to each sample to be filtered in the current block, the filtered prediction for the current block is obtained, which can be used to generate the final prediction for the current block.

[0053] In one embodiment, the filter shape uses a pattern (excluding the position (x, y)) around the location (x, y) of the sample to be filtered for prediction. The pattern is any subset of an MxN region or an MxN region around the sample to be filtered for prediction, where M is a first predefined positive number greater than 1 and N is a second predefined positive number greater than 1. Figure 6 An example of an MxN region is shown.

[0054] In one sub-implementation, M and N can be the same or different. For example, M and / or N are equal to 1, 2, 3, 4, ..., or any number specified in the standard. Another example is that M is greater than N if the current block's width is greater than its height. Another example is that N is greater than M if the current block's height is greater than its width. Yet another example is that M and / or N are adaptive based on position (x, y).

[0055] In another sub - embodiment, a subset of the MxN region can be the MxN region (a) excluding regions with horizontal coordinates > x and / or (b) excluding regions with vertical coordinates > y. In this example, the subset of MxN is the MxN region with (a)+(b), and M is equal to N. Thus, the MxN region is a square. The proposed method for the subset of the MxN region is not limited to being used when M equals N, and / or can be applied to cases where M > N and / or N > M. In another example, the subset of MxN is the MxN region with (a)+(b) and M > N. In another example, the subset of MxN is the MxN region with (a)+(b) and N > M.

[0056] An example of an MxN region (square) excluding regions with horizontal coordinates > x and vertical coordinates > y is shown in Figure 7 . An example of an MxN region (non - square, M < N) excluding regions with horizontal coordinates > x and vertical coordinates > y is shown in Figure 8 . An example of an MxN region (non - square, M > N) excluding regions with horizontal coordinates > x and vertical coordinates > y is shown in Figure 9 .

[0057] In another embodiment, multiple filter shapes are supported, and the selection from the multiple filter shapes depends on explicit signaling at the block level, CTU level, slice level, tile level, SPS level, PPS level, image level (e.g., image header), and / or sequence level. If the total number of candidate filter shapes in the list containing multiple filter shapes is K, a syntax signal for selecting the filter shape is issued. For example, the syntax is truncated unary coding and decoding. The shortest codeword is used to indicate the first candidate filter shape in the list.

[0058] In another embodiment, multiple determinations of filter parameters are supported, and the selection from the multiple determinations of filter parameters depends on explicit signaling at the block level, CTU level, slice level, tile level, SPS level, PPS level, image level (e.g., image header), and / or sequence level.

[0059] In another embodiment, the determination of filter parameters can be a derivation path of "deriving filter parameters using the template (neighboring region) of the current block" and / or an inheritance path of "inheriting filter information from previously decoded blocks". In a sub - embodiment, the selection between deriving filter parameters using the template (neighboring region) of the current block and inheriting filter information from previously decoded blocks depends on explicit signaling at the block level, CTU level, slice level, tile level, SPS level, PPS level, image level (e.g., image header), and / or sequence level.

[0060] In one sub-implementation, in response to deriving filter parameters using a template (neighborhood region) of the current block, the gold standard (i.e., the target for comparison) is the reconstructed sample on the current block template, with the goal of matching the filtered result to the gold standard on the template. Any method capable of minimizing the difference between the gold standard and the filtered result can be employed to obtain filter parameters that achieve a good match between the filtered result and the gold standard. For example, regression-based methods, such as Gaussian elimination, and / or any regression method consistent with the method used to derive a cross-component chromaticity model can be used. For example, chromaticity component samples can be predicted from the corresponding reconstructed luminance samples. The model parameters can be derived by minimizing the regression error between neighboring luminance and chromaticity samples.

[0061] In another sub-implementation, in response to deriving filter parameters using the template (neighborhood region) of the current block, the template refers to the top template, left template, top-left template, and / or any combination of the above templates, such as Figure 10 As shown. The selection of a template from multiple candidate templates can depend on the width, height, area, and / or position of the current block. For example, if the current block is located at the top of the image boundary and / or the top of the CTU row boundary, the top template and / or the top-left template are not used to derive filter parameters. Similarly, if the current block is located to the left of the image boundary and / or to the left of the CTU boundary, the left template and / or the top-left template are not used to derive filter parameters. In another example, different templates for deriving filter parameters refer to multiple determinations of filter parameters. The selection from multiple determinations of filter parameters depends on explicit communication at the block level, CTU level, slice level, tile level, SPS level, PPS level, image level (e.g., image header), and / or sequence level. For example, the template size depends on the block position. When the block is located at the right edge of the image or any predefined boundary, the top template does not include the template area outside the current image or any predefined range. When the block is located at the bottom edge of the image or any predefined boundary, the left template does not include the template area outside the current image or any predefined range.

[0062] In another sub-implementation, in response to inheriting filter information from previously encoded blocks, the filter information of previously encoded blocks is used to determine the filters for the current block. Filter information includes filter shape, filter parameters, partial filter parameters, or any combination thereof. Previously encoded blocks may be located in regions reconstructed prior to the current block within the same image (or frame) as the current block, or they may be located in image regions reconstructed prior to the current image (e.g., co-located images of the current block). For example, filter information is inherited through spatially adjacent candidates (including left-side and / or upper-side adjacent blocks) and / or spatially non-adjacent candidates, and / or history-based candidates, and / or temporal candidates, and / or propagation candidates. Spatially adjacent candidates may come from the left, upper, upper-left, upper-right, and lower-left adjacent blocks of the current block, any subset of these locations, or any combination thereof.

[0063] Spatially non-proximity candidates can come from any predefined location in the search pattern surrounding the current block, and / or any subset of such locations. History-based candidates can come from a history buffer that stores filter information for previous codec blocks. The history buffer is empty at predefined trigger points (schedules). For example, the history buffer is empty at the beginning or end of a slice, CTU / CTB, CTU / CTB line, image, tile, sequence, and / or any predefined unit. Temporal candidates can come from a buffer that stores filter information at a referenced location in a reference frame (or reference image) and / or a predefined co-location image, and / or filter information at any predefined location close to the referenced location. For example, the referenced location is a co-location block in a co-location image predicted intra-frame. As another example, the referenced location is indicated using motion information from neighboring blocks or any predefined blocks associated with the current block.

[0064] Propagation candidates can come from filter information at one or more reference locations referenced by motion information of neighboring blocks or any predefined blocks associated with the current block.

[0065] In one scenario, a predefined order is specified to examine all or any subset of the aforementioned inherited candidates. If a candidate cannot find filter information, it is skipped. The first available filter information is used for the current block. In another scenario, a list containing multiple inherited candidates is created. For example, the list size is fixed to a predefined number specified in the standard. A predefined order is set, inserting all or any subset of the aforementioned inherited candidates into the list. If not enough filter information can be found in all or any subset of the aforementioned inherited candidates, preset filter information is inserted. Again, for example, the list size can be adaptive based on how much filter information can be found using all or any subset of the aforementioned inherited candidates.

[0066] One way to select inherited filter information from a list is as follows. If the total number of inherited candidates in the list is K, a syntax is communicated to select inherited filter information from either the original list or a reordered list. For example, the syntax is a truncated unary codec. The shortest codeword is used to indicate the first candidate in the list. An example of reordering the list is as follows. The list is reordered (or rearranged) based on the measurement results of each candidate on the template. For example, the measurement of a candidate depends on the distortion between the reconstructed sample and the predicted sample of the template, which is produced by applying the filter information of that candidate to the template. Candidates with smaller distortion on the template can be considered promising candidates during the measurement process. Promising candidates are reordered and placed at the front of the list. One change is that the syntax does not need to indicate candidates from the reordered list. During measurement, only the most promising candidates are retained, and after checking each candidate in the list, the most promising candidate with the smallest distortion is selected for the current block.

[0067] In another embodiment, when generating the filterable prediction sample for the top-left corner sample of the current block, all input samples for filtering use the reconstructed sample that is adjacent to the current block.

[0068] When generating the predicted samples to be filtered near the top and / or left boundary of the current block, a portion of the input samples to be filtered uses reconstructed samples adjacent to the current block, and a portion of the input samples to be filtered uses previously predicted samples within the current block. When generating the predicted samples to be filtered inside the current block and / or away from the top / left boundary of the current block, all input samples to be filtered use previously predicted samples from the current block.

[0069] Figure 11 An example of a sample to be predicted is shown near the top boundary of the current block. Figure 12 An example is shown for the sample to be predicted in the frame portion of the current block and / or away from the top and left boundaries.

[0070] In another embodiment, a filter-based target intra-frame prediction mode is used to generate the luma predictor. Therefore, the current block refers to the luma component. In one example using a single-tree structure, the current block is the luma codec block (CB) in a codec unit, which consists of one or more luma CBs and one or more chroma CBs. In another example using a dual-tree structure, the current block is the luma codec block (CB) in a codec unit, which consists of one or more luma CBs.

[0071] In another embodiment, a filter-based target intra-frame prediction mode is used to generate chroma predictors. Therefore, the current block refers to one or more chroma components, such as Cb and / or Cr. For an example using a single-tree structure, the current block is a chroma codec block (CB) within a codec unit consisting of one or more luma CBs and one or more chroma CBs. For another example using a dual-tree structure, the current block is a chroma codec block (CB) within a codec unit consisting of one or more chroma CBs.

[0072] (2) Target Filter-Based Novel Prediction Mode When the current block belongs to a non-intra-frame coding mode, the block vector information (including the block vector) and / or motion vector information (including the motion vector and / or the reference image) points to the reference block. If the non-intra-frame coding mode of the current block is inter-frame, the motion information of the current block points to the reference block in the reference image, or to the reference block in any predefined co-position image of the current image (which is different from the current image). If the non-intra-frame coding mode of the current block is intra-block copy (IBC) and / or intra-TMP, which searches within a predefined region and finds a reference block (whose template matches the template of the current block well) to predict the current block, the block vector information of the current block points to the reference block in the current image. Similar to the filter-based target intra-frame prediction mode, the filter (including the determination of the filter shape and / or filter parameters) determines the reference block of the current block. In one embodiment, for the filter of the reference block, the filter parameters can be determined by using a template (neighboring region) of the reference block, i.e., applying the derivation process in the derivation path to the template of the reference block instead of the current block; and / or by inheriting filter information from a previous codec block, i.e., applying the inheritance process in the inheritance path to the reference block instead of the current block. For example, spatially adjacent or non-adjacent candidates in the list can come from blocks that are adjacent or non-adjacent to the reference block. Then, the filter prediction of the reference block (denoted as filter_pred_ref) is derived using the filter of the reference block, which can be used to generate the final prediction for the current block. The details of deriving filter_pred_ref on the reference block are consistent with the way the filter is applied to each sample to be filtered in the current block in Section (1), the main difference being that the operation here is performed on the reference block.

[0073] In one embodiment, a residual (R) for the reference block is generated by comparing `filter_pred_ref` with reconstructed samples of the reference block, and this residual is used to correct the filtered prediction of the current block obtained using a filter-based target intra-prediction mode. For example, the residual is calculated as the difference between `filter_pred_ref` and the reconstructed samples of the reference block, and the filtered prediction from the filter-based target intra-prediction mode is adjusted by adding or subtracting this residual. This adjusted filtered prediction is used to generate the final prediction for the current block.

[0074] In another embodiment, the filtered prediction of the current block obtained using the filter-based target intra-frame prediction mode is combined with filter_pred_ref, and the combined filtered prediction is used to generate the final prediction of the current block, such as... Figure 13 As shown.

[0075] (3) Obtain an efficient filter The efficient filter for the current block can be obtained from previous blocks. In one embodiment, such as... Figure 14 As shown in process 1420, for a reconstruction block preceding the current block, a filter is derived between the prediction 1422 and the reconstruction 1424 for that reconstruction block (in process 1426). In response to deriving filter parameters using the reconstruction block, `golden` represents the reconstructed samples on that block, with the goal of matching the filtered result to `golden`. For example, applying a filter to valid predicted samples (marked with slashes) near position (i,j) 1440 aims to derive filter parameters to minimize the difference between the filtered result (marked with backslashes) 1450 and the reconstructed value at position (i,j), where (i,j) can be any position within the current block. Any method can be used to minimize the difference between `golden` and the filtered result to obtain filter parameters whose filtered result matches `golden` well. For example, regression-based methods (e.g., Gaussian elimination), and / or arbitrary regression methods (e.g., cross-component chroma modes or arbitrary cross-component chroma modes in the standard) consistent with methods used to derive cross-component chroma modes, and / or methods used to derive filter-based target intra-prediction modes or novel prediction modes can be used. The derived filter is stored for the reconstructed block in filter buffer 1430. Subsequently, as shown in flow 1410, the current block can obtain the stored filter of the reconstructed block through an inheritance path, and / or use the obtained filter to generate the filtered prediction 1414 of the current block.

[0076] In one approach, when generating the filtered prediction for the current block, the filter input is the initial predictor for the current block. The initial prediction 1412 can refer to intra-frame prediction or non-intra-frame prediction, such as a motion-compensated predictor or a block vector predictor. In another approach, when generating the filtered prediction for the current block, the filter input is a reconstructed sample or a predicted sample preceding the current sample to be filtered. The initial prediction can refer to intra-frame prediction or non-intra-frame prediction, such as a motion-compensated predictor or a block vector predictor. In one embodiment, there are some constraints on deriving the filter on the reconstructed block. These constraints may depend on the encoding / decoding mode, block location, block width, block height, and / or block area of ​​the reconstructed block. For example, if the reconstructed block is not intra-frame encoded, an efficient filter is not derived on that reconstructed block. As another example, if the width, height, and / or area of ​​the reconstructed block is less than a predefined threshold, an efficient filter is not derived on that reconstructed block. In another embodiment, a reconstructed block may refer to a CB / CU, TB / TU, any predefined region, each CB / CU, or each TB / TU.

[0077] The efficient filter for the current block can be obtained from the template of the previous block. In one embodiment, for a reconstructed block preceding the current block, a filter between the prediction and reconstruction of the template of that reconstructed block is derived. In response to deriving filter parameters using the template of the reconstructed block, `golden` is the reconstructed sample on the template of that reconstructed block, with the goal of matching the filtered result with `golden`. Any method can be used to minimize the difference between `golden` and the filtered result to obtain filter parameters whose filtered result matches `golden` well. For example, regression-based methods (e.g., Gaussian elimination), arbitrary regression methods (e.g., cross-component chroma modes and / or arbitrary cross-component chroma modes in the standard) consistent with methods used to derive cross-component chroma modes, and / or methods used to derive filters for a target intra-prediction mode or a novel prediction mode based on filters can be used. The derived filter is stored for that reconstructed block. Subsequently, the current block can obtain the stored filter of the reconstructed block (derived on the template of that reconstructed block) via an inheritance path, and / or use the obtained filter to generate the filtered prediction for the current block.

[0078] When generating the filter prediction for the current block in Section (1), the output filter result at one location can be used as the filter input at another location in the current block. In one embodiment, there are some constraints on deriving the filter on the template of the reconstructed block. These constraints may depend on the encoding / decoding mode, block location, block width, block height, and / or block area of ​​the reconstructed block. For example, if the reconstructed block is not intra-coded, an efficient filter is not derived on the template of the reconstructed block. As another example, if the width, height, and / or area of ​​the reconstructed block are less than a predefined threshold, an efficient filter is not derived on the template of the reconstructed block. In another embodiment, the reconstructed block may refer to a CB / CU, TB / TU, any predefined region, each CB / CU, or each TB / TU.

[0079] In another embodiment, such as Figure 15As shown, the efficient filter for the current block can be obtained from adjacent and / or non-adjacent reference regions of the current block. Adjacent reference regions 1510 can be any reference region in the current image that was reconstructed before the current block and is adjacent to the current block. Non-adjacent reference regions 1520 can be any reference region in the current image that was reconstructed before the current block and is not adjacent to the current block, or any reference region in any predefined reference image (different from the current image). A sub-region 1530 within the reference region is accessed to derive the efficient filter. Subsequently, the current block can obtain the derived filter from this sub-region. The goal of this derivation is to enable the derived filter to produce an output that well matches the reconstruction of the sub-region. In response to deriving filter parameters using the sub-region, `golden` is the reconstructed sample on the sub-region, with the goal of matching the filtered result to `golden`. Any method can be used to minimize the difference between `golden` and the filtered result to obtain filter parameters whose filtered result well matches `golden`. For example, regression-based methods (e.g., Gaussian elimination), arbitrary regression methods consistent with methods used to derive cross-component chroma modes (e.g., cross-component chroma modes in the standard and / or arbitrary cross-component chroma modes), and / or methods for deriving filter-based target intra-prediction modes or novel prediction modes can be used. Subsequently, the current block uses the obtained filters to generate filtered predictions for the current block.

[0080] In one approach, when generating the filtered prediction for the current block, the filter input is the initial predictor for the current block. The initial prediction can refer to intra-frame prediction or non-intra-frame prediction, such as a motion-compensated predictor or a block vector predictor. In another approach, when generating the filtered prediction for the current block, the filter input is a reconstructed sample or a predicted sample preceding the current sample to be filtered. In one embodiment, determining a sub-region is based on certain conditions. The determination of a sub-region may depend on its encoding / decoding mode, block location, block width, block height, and / or block area. For example, if a sub-region is intra-frame encoded, it can be used to derive an efficient filter. As another example, if a sub-region has a width, height, and / or area greater than a predefined threshold, it can be used to derive an efficient filter.

[0081] (4) Storage rules for the current block's filter-based target intra-frame prediction mode or target filter-based novel prediction mode. The current block stores all or any subset of filter information {filter shape and / or filter parameters}, and / or the current block can reference all or any subset of filter information {filter shape and / or filter parameters}. For example, the stored filter information of the current block can be inherited by subsequent blocks in the inheritance path of subsequent blocks. The stored filter information can be obtained from efficient filters using the method proposed in Section (3).

[0082] The method proposed in this invention can be enabled and / or disabled based on implicit rules (e.g., block width, height, or area), or based on explicit rules (e.g., block-level, tile-level, slice-level, image-level, SPS-level, or PPS-level syntax). For example, the proposed method is applied when the block area is less than / greater than a threshold.

[0083] In this invention, the term "block" may refer to TU / TB, CU / CB, PU / PB, predefined area, or CTU / CTB.

[0084] The methods proposed in this invention can be applied in any combination.

[0085] Any of the aforementioned filter derivation methods can be implemented in the encoder and / or decoder. For example, any of the aforementioned methods can be implemented in the inter-frame and / or intra-frame and / or IBC and / or prediction and / or transform modules of the encoder, and / or in the inter-frame and / or intra-frame and / or IBC and / or prediction and / or transform modules of the decoder. Alternatively, any of the aforementioned methods can be circuitically coupled to the inter-frame and / or intra-frame and / or IBC and / or prediction and / or transform modules of the encoder, and / or coupled to the inter-frame and / or intra-frame and / or IBC and / or prediction and / or transform modules of the decoder, to provide the information required by the inter-frame and / or intra-frame and / or IBC and / or prediction and / or transform modules. (Refer to...) Figure 1A and Figure 1B The encoder and decoder in the code, any of the aforementioned methods can be used in the encoder's inter-frame / intra-frame / prediction / transform module (e.g. Figure 1A Implemented in the intra-frame prediction unit 110 and / or inter-frame prediction unit 112 of the decoder, and / or in the inter-frame / intra-frame / prediction / transform module (e.g. Figure 1B Implemented in the intra-frame prediction unit 150 and / or MC unit 152.

[0086] Figure 16A flowchart illustrating an exemplary video codec system according to an embodiment of the present invention is provided, which uses the residual of a reference block to correct the filtered prediction of the current block. The steps shown in the flowchart can be implemented as program code executable on one or more processors (e.g., one or more CPUs) at the encoder and / or decoder ends. The steps shown in the flowchart can also be implemented in hardware, such as one or more electronic devices or processors configured to perform the steps in the flowchart. According to the method, in step 1610, input data associated with the current block is received, wherein the input data includes pixel data to be encoded at the encoder end, or encoded data associated with the current block to be decoded at the decoder end. In step 1620, a reference block for the current block is determined. In step 1630, a residual of the reference block is determined, the residual corresponding to the difference between a filtered prediction sample of the reference block and a reconstructed sample of the reference block, wherein the filtered prediction sample of the reference block is generated by applying a filter-based target intra-prediction mode to the reference block. In step 1640, based on the residual of the reference block, the filter-based target intra-prediction associated with the filter-based target intra-prediction mode is corrected to a target-modified filter-based prediction for the current block. In step 1650, a corrected current filtered prediction is generated by applying the modified filter-based prediction for the current block to the current block.

[0087] Figure 17 A flowchart of an exemplary video encoding / decoding system according to an embodiment of the present invention is shown. The system uses filter parameters derived and stored by matching filtering results with reconstructed samples of a previously reconstructed block. According to the method, in step 1710, input data associated with the current block is received, wherein the input data includes pixel data to be encoded at the encoder end, or encoded data associated with the current block to be decoded at the decoder end. In step 1720, stored filter parameters associated with a target filter of a filter-based prediction mode are determined, wherein the stored filter parameters are derived by matching filtering results with reconstructed samples of a previously reconstructed block, and the filtering results are generated by applying the target filter to the prediction of the previous block. In step 1730, the target filter using the stored filter parameters is applied to the initial predictor of the current block to generate a current predictor for the current block. In step 1740, the current block is encoded or decoded using encoding / decoding information including the current predictor.

[0088] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. Those skilled in the art can practice the invention by modifying each step, rearranging the steps, splitting the steps, or combining the steps without departing from the spirit of the invention. Specific syntax and semantics are used in the disclosure to illustrate examples of implementing the invention. Those skilled in the art can practice the invention by substituting equivalent syntax and semantics without departing from the spirit of the invention.

[0089] The foregoing description is intended to enable those skilled in the art to practice the invention according to specific applications and requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but rather to be given the broadest scope, consistent with the principles and novel features disclosed herein. In the foregoing detailed description, various specific details have been shown to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced.

[0090] The embodiments of the invention described above can be implemented in various ways, in software code, or in a combination of both. For example, one embodiment of the invention may be one or more circuits integrated into a video compression chip or program code integrated into video compression software to perform the processes described herein. Another embodiment of the invention may be program code to be executed on a digital signal processor (DSP) to perform the processes described herein. The invention may also relate to multiple functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). These processors may be configured to perform specific tasks according to the invention, defining the specific methods embodied in the invention by executing machine-readable software code or firmware code. The software code or firmware code may be developed in different programming languages ​​and in different formats or styles. The software code may also be compiled for different target platforms. However, different code formats, styles, and languages ​​of the software code, as well as other means of configuring the code to conform to the tasks of the invention, do not depart from the spirit and scope of the invention.

[0091] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered illustrative in all respects only, and not restrictive. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All variations within the meaning and equivalence of the claims should be included within their scope.

Claims

1. A video encoding / decoding method, comprising: Receive input data related to the current block, wherein the input data includes pixel data to be encoded at the encoder end, or encoded data related to the current block to be decoded at the decoder end; Determine the reference block for the current block; Determine the residual of the reference block, which corresponds to the difference between the filtered prediction sample of the reference block and the reconstructed sample of the reference block, wherein the filtered prediction sample of the reference block is generated by applying a filter-based target intra-frame prediction mode to the reference block; Based on the residual of the reference block, the filter-based target intra-prediction associated with the filter-based target intra-prediction mode is modified into a modified filter-based prediction for the current block. as well as A modified current filter prediction is generated by applying the modified filter-based prediction for the current block to the current block.

2. The method of claim 1, wherein the filter-based target intra-frame prediction mode corresponds to EIP (Extrapolation Filter-based Intra-frame Prediction).

3. The method of claim 1, wherein the modified filter-based prediction for the current block is determined by subtracting the residual from the current prediction or adding the residual to the current prediction.

4. The method of claim 1, wherein the modified current filtered prediction is combined with the filtered prediction sample of the reference block to generate the final prediction of the current block.

5. The method of claim 1, wherein the information related to the filter-based target intra-frame prediction mode includes filter shape and filter parameters.

6. A video encoding / decoding apparatus, comprising one or more electronic devices or processors, configured as follows: Receive input data related to the current block, wherein the input data includes pixel data to be encoded at the encoder end, or encoded data related to the current block to be decoded at the decoder end; Determine the reference block for the current block; Determine the residual of the reference block, which corresponds to the difference between the filtered prediction sample of the reference block and the reconstructed sample of the reference block, wherein the filtered prediction sample of the reference block is generated by applying a filter-based target intra-frame prediction mode to the reference block; Based on the residual of the reference block, the filter-based target intra-prediction associated with the filter-based target intra-prediction mode is modified into a modified filter-based prediction for the current block. as well as A modified current filter prediction is generated by applying the modified filter-based prediction for the current block to the current block.

7. A video encoding / decoding method, comprising: Receive input data related to the current block, wherein the input data includes pixel data to be encoded at the encoder end, or encoded data related to the current block to be decoded at the decoder end; Determine stored filter parameters associated with a target filter for a filter-based intra-frame prediction mode, wherein the stored filter parameters are derived by matching the filtering result with reconstructed samples of a previous reconstructed block, and the filtering result is generated by applying the target filter to the prediction of the previous block. The target filter, using the stored filter parameters, is applied to the initial predictor of the current block to generate the current predictor of the current block; as well as The current block is encoded or decoded using encoding / decoding information, including the current predictor.

8. The method of claim 7, wherein the target filter of the filter-based intra-prediction mode corresponds to EIP (Extrapolation Filter-based Intra-prediction).

9. The method of claim 7, wherein the initial predictor is generated by intra-frame prediction or non-intra-frame prediction.

10. The method of claim 7, wherein the initial predictor corresponds to a reconstructed sample or a predicted sample preceding the current sample to be filtered.

11. The method of claim 7, wherein the previously reconstructed block corresponds to one or more templates of one or more previous blocks.

12. The method of claim 7, wherein the previously reconstructed block corresponds to an adjacent reference region or a non-adjacent region.

13. A video encoding / decoding apparatus, comprising one or more electronic devices or processors, configured as follows: Receive input data related to the current block, wherein the input data includes pixel data to be encoded at the encoder end, or encoded data related to the current block to be decoded at the decoder end; Determine stored filter parameters associated with a target filter for filter-based intra-frame prediction, wherein the stored filter parameters are derived by matching the filtering result with reconstructed samples of a previous reconstructed block, and the filtering result is generated by applying the target filter to the prediction of the previous block. The target filter, using the stored filter parameters, is applied to the initial predictor of the current block to generate the current predictor of the current block; and The current block is encoded or decoded using encoding / decoding information, including the current predictor.

Images