Combination of extrapolation filter based intra prediction and other intra prediction

Abstract

In one implementation, we propose to improve the intra prediction performance by blending the prediction from EIP (extrapolation filter based intra prediction) with the prediction from EIP or non-EIP (directional, planar, DC intra prediction). In one example, this is done by employing N regular intra predictors as well as the EIP prediction. Template analysis can be performed to select the modes to blend and the corresponding weights. In one implementation, we specifically propose to employ DIMD and TIMD template analysis. That is, EIP with DIMD and EIP with TIMD are used to combine EIP with other intra prediction modes.

Description

[0001] Cross-reference to related applications This application claims priority to European application number 24305054.9, filed on January 9, 2024, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] This embodiment generally relates to methods and apparatus for performing intra-frame prediction in video encoding and decoding. Background Technology

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transform to fully utilize the spatial and temporal redundancy in video content. Generally, intra-frame or inter-frame prediction is used to leverage intra- or inter-frame correlations, followed by transform, quantization, and entropy coding of the differences between the original and predicted blocks (often represented as prediction error or prediction residuals). To reconstruct the video, the compressed data is decoded through the inverse processes corresponding to entropy coding, quantization, transform, and prediction. Summary of the Invention

[0004] According to one embodiment, a video decoding method is proposed, comprising: generating a first prediction of a block of an image based on an intra-frame prediction mode based on an extrapolation filter; generating a second prediction for the block based on another intra-frame prediction mode; mixing the first prediction and the second prediction to form a prediction for the block; and decoding the block based on the prediction for the block.

[0005] According to another embodiment, a video coding method is proposed, comprising: generating a first prediction of a block of an image based on an intra-frame prediction mode based on an extrapolation filter; generating a second prediction for the block based on another intra-frame prediction mode; mixing the first prediction and the second prediction to form a prediction for the block; and encoding the block based on the prediction for the block.

[0006] According to another embodiment, an apparatus for video decoding is proposed, comprising at least one memory and one or more processors, wherein the one or more processors are configured to: generate a first prediction for a block of an image based on an intra-frame prediction mode based on an extrapolation filter; generate a second prediction for the block based on another intra-frame prediction mode; mix the first prediction and the second prediction to form a prediction for the block; and decode the block based on the prediction for the block.

[0007] According to another embodiment, an apparatus for video coding is proposed, comprising at least one memory and one or more processors, wherein the one or more processors are configured to: generate a first prediction of a block of an image based on an intra-frame prediction mode based on an extrapolation filter; generate a second prediction for the block based on another intra-frame prediction mode; mix the first prediction and the second prediction to form a prediction for the block; and encode the block based on the prediction for the block.

[0008] One or more embodiments also provide a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding or decoding method according to any of the embodiments described herein. One or more of these embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to the methods described herein.

[0009] One or more embodiments also provide a computer-readable storage medium storing video data generated according to the methods described above. One or more embodiments also provide a method and apparatus for transmitting or receiving video data generated according to the methods described herein. Attached Figure Description

[0010] Figure 1 The diagram illustrates a block diagram of a system in which various aspects of this embodiment can be implemented.

[0011] Figure 2 A block diagram illustrating an embodiment of a video encoder is shown.

[0012] Figure 3 A block diagram illustrating an embodiment of a video decoder is shown.

[0013] Figure 4 The illustration shows the candidates for spatial GPM (Geometric Partition Pattern).

[0014] Figure 5 The image shows the GPM template.

[0015] Figure 6 The diagram illustrates three defined filter shapes with fifteen inputs and one output.

[0016] Figure 7 The diagram illustrates three types of refactoring regions used for EIP.

[0017] Figure 8 The illustration shows an example of generating predictions for different positions in the current block in diagonal order.

[0018] Figure 9The illustration depicts a method, according to an embodiment, for generating a prediction for the current block by mixing the EIP prediction of the current block with several other intra-frame predictions of the current block. Detailed Implementation

[0019] Figure 1 The diagram illustrates an example of a system in which various aspects and embodiments can be implemented. System 100 may be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100 may be embodied individually or in combination in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described in this application.

[0020] System 100 includes at least one processor 110 configured to execute instructions loaded thereon to implement various aspects, such as those described in this application. Processor 110 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 140 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0021] System 100 includes an encoder / decoder module 130 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, as is known to those skilled in the art, the encoder / decoder module 130 may be implemented as a separate element of system 100, or it may be incorporated within processor 110 as a combination of hardware and software.

[0022] Program code to be loaded onto processor 110 or encoder / decoder 130 to execute the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0023] In several embodiments, the memory within processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 110 or encoder / decoder module 130) is used for one or more of these functions. External memory may be memory 120 and / or storage device 140, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as working memory for MPEG-2, HEVC, or VVC.

[0024] Inputs to the components of system 100 can be provided by various input devices, as indicated in block 105. Such input devices include, but are not limited to: (i) an RF section that receives, for example, RF signals transmitted over the air by a broadcasting device, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0025] In various embodiments, as known in the art, the input device of block 105 has associated respective input processing elements. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) band-limiting it again to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, down-converters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0026] Furthermore, the USB and / or HDMI terminals may include their respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or within processor 1010. Similarly, various aspects of USB or HDMI interface processing can be implemented as needed, either within a separate interface IC or within processor 1010. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 110 and encoder / decoder 130, to operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0027] Various components of system 100 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data between them using a suitable connection arrangement 1140 (e.g., internal buses as known in the art, including I2C buses, wiring, and printed circuit boards).

[0028] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 190 may be implemented, for example, in a wired and / or wireless medium.

[0029] In various embodiments, data is streamed to system 100 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals in these embodiments are received via a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data via an HDMI connection to input block 105 to provide streaming data to system 100. Still other embodiments use an RF connection to input block 105 to provide streaming data to system 100.

[0030] System 100 can provide output signals to various output devices, including display 165, speaker 175, and other peripheral devices 185. In various examples of embodiments, other peripheral devices 185 include one or more of a standalone DVR, disc player, stereo system, lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, signaling such as AV link, CEC, or other communication protocols that enable inter-device control with or without user intervention is used to communicate control signals between system 100 and display 165, speaker 175, or other peripheral devices 185. Output devices can be communicatively coupled to system 100 via dedicated connections through their respective interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speaker 175 can be integrated into a single unit with other components of system 100 in electronic devices, such as televisions. In various embodiments, display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0031] For example, if the RF portion of input 105 is part of a standalone set-top box, then display 165 and speaker 175 may alternatively be separate from one or more other components. In various embodiments where display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0032] Figure 2 The illustration shows an example video encoder 200, such as a VVC (Variety Video Coding) encoder. Figure 2 It can also be shown that the encoder is an improvement on the VVC standard or an encoder that uses a similar technology to VVC.

[0033] In this application, the terms "reconstructed" and "decoded" are used interchangeably, as are the terms "encoded" and "coded," and the terms "image," "picture," and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side and "decoded" is used on the decoder side.

[0034] Before being encoded, the video sequence may undergo pre-coding (201), for example, by applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or by performing remapping of the input image components to obtain a more resilient signal distribution to compression (e.g., using histogram equalization with one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0035] In encoder 200, as described below, the image is encoded by encoder elements. The image to be encoded is segmented (202) and processed in units such as CUs (coding units). Each unit is encoded using, for example, an intra-frame mode or an inter-frame mode. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (260). In inter-frame mode, motion estimation (275) and compensation (270) are performed. The encoder determines (205) which of the intra-frame or inter-frame modes is used to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (210) the prediction block from the original image block.

[0036] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with motion vectors and other syntax elements (such as image segmentation information), are entropy encoded (245) to output a bitstream. As a non-limiting example, context-based adaptive binary arithmetic coding (CABAC) can be used to encode syntax elements into a bitstream.

[0037] The encoder can skip the transform and directly apply quantization to the untransformed residual signal. The encoder can bypass both the transform and quantization, that is, directly encode the residual without applying either the transform or quantization process.

[0038] The encoder decodes the encoded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inversely transformed (250) to decode the prediction residuals. The image blocks are reconstructed by combining (255) the decoded prediction residuals and the prediction blocks. An in-loop filter (265) is applied to the reconstructed image to perform, for example, deblocking / SAO (Sampling Adaptive Offset) / AFL (Adaptive Loop Filter) filtering to reduce coding artifacts. The filtered image is stored in a reference image buffer (280).

[0039] Figure 3 A block diagram of an example video decoder 300 is shown. In decoder 300, as described below, the bitstream is decoded by decoder elements. Video decoder 300 typically performs operations similar to... Figure 2 The decoding traversal that is the inverse of the encoding traversal (pass) described in the text, such as Figure 2 As described in [the document]. Encoder 200 typically also performs video decoding as part of the encoding of video data.

[0040] Specifically, the input to the decoder includes a video bitstream, which can be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, prediction modes, motion vectors, and other encoded information. Image segmentation information indicates how the image is segmented. Therefore, the decoder can segment (335) the image based on the decoded image segmentation information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residuals. By combining (355) the decoded prediction residuals and prediction blocks, image blocks are reconstructed. Prediction blocks can be obtained (370) from intra-frame prediction (360) or motion-compensated prediction (i.e., inter-frame prediction) (375). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored at a reference image buffer (380). Note that for a given image, the contents of the reference image buffer 380 on the decoder 300 side are exactly the same as the contents of the reference image buffer 280 on the encoder 200 side for the same image.

[0041] The decoded image can undergo further post-decoding processing (385), such as inverse color transformation (e.g., conversion from YCbCr4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in pre-encoding processing (201). Post-decoding processing can utilize metadata derived in pre-encoding processing and signaled in the bitstream.

[0042] Intra-frame prediction based on extrapolation filters (EIP) is a novel coding tool being investigated in exploratory experiments (see L. Xu et al., “EE2-2.7: Intra-frame prediction mode based on extrapolation filters,” document JVET-AF0080, Joint Video Experts Group (JVET), 32nd meeting, Hanover, Germany, October 13-20, 2023, hereinafter referred to as “JVET-AF0080”). It is based on learning extrapolation filters from a reconstructed template and applying the learned extrapolation filters to the current block to generate the predicted signal. Although this tool provides significant coding gain, it remains underutilized. For example, compared to conventional intra-frame prediction modes, this mode is used as a single predictor. That is, it is not used / mixed with other intra-frame prediction modes. To improve prediction quality and thus coding gain, this embodiment proposes to overcome this limitation by proposing a new hybrid mode that combines EIP with other EIPs and / or other intra-frame prediction modes.

[0043] Decoder-side intra-frame mode derivation (DIMD) In ECM (Enhanced Compression Model), DIMD derives indices of N (≤ 5) intra-prediction modes from the gradient in the template of the decoding reference sample of the current luminance CB to be encoded / decoded. These intra-prediction modes are likely the best intra-prediction modes for predicting the current luminance CB in terms of rate distortion.

[0044] As described in M. Coban et al.'s paper "Algorithmic Description of Enhanced Compression Model 10 (ECM 10)," document JVET-AE2025, ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, Joint Video Experts Group (JVET), 31st Meeting, Geneva, Switzerland, July 11-19, 2023 (hereinafter referred to as "JVET-AE2025"), when decoder-side intra-frame mode derivation (DIMD) is applied, up to five intra-frame modes are derived from the reconstructed neighbor samples, and those five predictors are combined with a planar mode predictor, the weights of which are derived from, as described in JVET-O0449 (see T. Guionnet et al.'s paper "Non-CE3: Decoder-Side Intra-Frame Mode Derivation Using Planar Predictive Fusion," document JVET-O0449-v2, ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG). The directional gradient histogram described in the 15th meeting of the Joint Video Experts Group (JVET) of 11: Gothenburg, Sweden, July 3-12, 2019 (hereinafter referred to as "JVET-O0449") is derived. The division operation in the weight derivation is performed using the same lookup table (LUT)-based integration scheme used in the Cross-Component Linear Model (CCLM). For example, the division operation in the direction calculation is computed by the following LUT-based scheme. : , in, .

[0045] For size If the magnitude of one of the upper and left histograms is twice that of the other, then the weights of each of the five derived patterns are modified. In this case, the weights are position-dependent and calculated as follows: If the histogram above is twice the size of the histogram on the left, then: .

[0046] If the magnitude of the left histogram is twice the magnitude of one of the upper histograms, then: , in It is the unmodified uniform weight of DIMD as selected in JVET-O0449. It is predefined and set to 10.

[0047] The exported intra-frame modes are included in the main list (MPM) of the most probable intra-frame modes, so the DIMD process is performed before the MPM list is constructed. The main exported intra-frame modes of the DIMD block are stored with the block and are used for the construction of the MPM list for adjacent blocks.

[0048] Finally, note that, depending on the availability of reconstructed samples, the regions used to compute the histogram of directional gradients for adjacent reconstructed samples are modified compared to the JVET-0449 method. If available, the region of the current WxH brightness CB decoding reference sample is extended to the upper right, by a maximum of W additional columns. If possible, it is extended to the lower left, by a maximum of H additional rows.

[0049] Template-based intra-frame mode derivation (TIMD) Like DIMD, TIMD follows a two-step process for encoding / decoding the current luminance CB: an intra-prediction mode index derivation step involving the template of the decoding reference sample of the current luminance CB, and a step of actually predicting the current luminance CB. The left side can be used... Part and current Brightness above CB Part of the template is generated. During the TIMD derivation step, the tested intra-prediction mode predicts the template for the current luminance CB from the template's decoding reference sample set.

[0050] As described in JVET-AE2025, for each intra-prediction mode in the MPM, and for the wide-angle mode (if top-right and / or bottom-left reference samples are available), the SATD (Sum of Absolute Transform Differences) between the predicted and reconstructed samples of the template is calculated. The two intra-prediction modes with the smallest SATD are selected as TIMD modes. These two TIMD modes (Mode 1 and Mode 2) are weighted and fused after applying the PDPC (Position-Related Intra-Prediction Combination) process, and such weighted intra-prediction is used to encode the current CU. The Position-Related Intra-Prediction Combination (PDPC) is included in the derivation of the TIMD modes.

[0051] Compare the costs of the two selected models: , in, It is the cost of Mode 1 (with minimum SATD), and This is the cost of Mode 2 (with a second minimum SATD). If this condition is true, then fusion is applied; otherwise, only Mode 1 is used.

[0052] The weights of the patterns are calculated based on their SATD costs as follows: .

[0053] Division operations are performed using the same lookup table (LUT)-based integration scheme as CCLM.

[0054] Intra-frame prediction fusion As described in JVET-AE2025, this intra-frame prediction method derives the prediction samples as a weighted combination of multiple predictors generated from different reference lines. In this process, multiple intra-frame predictors are generated and then fused by a weighted average. The process of deriving the predictors to be used in the fusion process is described below: 1) For single-mode angular intra-prediction modes including TIMD and DIMD, the proposed method derives intra-prediction by weighting intra-prediction obtained from multiple reference lines, expressed as: ,in It is an intra-frame prediction from the default reference line, and This is a prediction from the line above the default reference line. The weights are set to... and .

[0055] 2) For TIMD modes with hybrid patterns, Used in the first mode ,and Used in the second mode .

[0056] 3) For DIMD patterns with mixing, the number of predictors selected for the weighted average is increased from 3 to 6.

[0057] When the intra-frame prediction mode has a non-integer slope (requiring reference sample interpolation) and the block size is greater than 16, the intra-frame prediction fusion method is applicable to luma blocks. It is used in conjunction with MRL (Multiple Reference Lines) and is not applicable to ISP-coded blocks. In the method studied in subtest a, PDPC is applied to the intra-frame prediction mode using the reference line closest to the current block.

[0058] Spatial Geometric Partitioning Pattern (SGPM).

[0059] Spatial Geometric Segmentation Mode (SGPM) is an intra-frame mode of an inter-frame coding tool similar to Geometric Segmentation Mode (GPM), where two prediction parts are generated from the intra-frame prediction process. In this mode, a candidate list is constructed, where each entry contains a segmentation split and two intra-frame prediction modes, such as... Figure 4 As shown, 26 segmentation modes and 3 intra-frame prediction modes are used to form a combination. The length of the candidate list is set to 16. The selected candidate index is notified by a signal.

[0060] Use such as Figure 5The template shown reorders the list, where the SAD (sum of absolute differences) between the template's predictions and reconstructions is used for sorting. The template size is fixed at 1.

[0061] For each segmentation pattern, an IPM list is derived for each segment using the same intra-to-out-of-frame GPM list derivation. The IPM list size is set to 3. In the list, the TIMD-derived pattern is replaced by two derived patterns in the horizontal and vertical directions.

[0062] The SGPM mode is suitable for restricted block sizes: 4 ≤ width ≤ 64, 4 ≤ height ≤ 64, width < height * 8, height < width * 8, width * height ≥ 32.

[0063] The PPS flag is encoded to indicate whether blending of two intra-frame predictions is not allowed. When this PPS flag is set to false, the following adaptive blending is also applied to the spatial GPM, where the blending depth τ is derived as follows: • If min(width, height) = 4, then choose 1 / 2τ. • Otherwise, if min(width, height) = 8, then choose τ. • Otherwise, if min(width, height) = 16, then choose 2τ. • Otherwise, if min(width, height) = 32, then choose 4τ. • Otherwise, choose 8τ.

[0064] Otherwise (with the PPS flag set to true), 1 / 4τ is always used for spatial GPM encoded blocks to ensure that blending is not used when SGPM blocks have perfectly horizontal or vertical split angles, and a much narrower blending width is used when SGPM blocks have other split angles. Note that this flag is set to true in the current Common Test Conditions (CTC) for screen content video.

[0065] Intra-frame prediction (EIP) based on extrapolation filters.

[0066] The extrapolation-based intra-prediction described in JVET-AF0080 is processed in three steps. First, the extrapolation filter coefficients are derived from the adjacent reconstruction regions of the current block or inherited from previous EIP blocks. Second, the extrapolation process generates the prediction signal from the top left to the bottom right corner within the current block. Third, the intra-prediction angle is derived by analyzing the gradient of the prediction block, and then the corresponding intra-prediction mode is used to select the MTS (Multiple Transform Selection), NSPT (Non-Separable Master Transform), and LFNST (Low-Frequency Non-Separable Transform) kernels for transformation.

[0067] EIP applications are limited to blocks no larger than 32x32 and luminance components.

[0068] Obtain the EIP filter.

[0069] Used Figure 6 The three EIP filter shapes are shown. The corresponding EIP modes are represented as full, horizontal, and vertical EIP modes, respectively.

[0070] There are two methods for obtaining the filter coefficients of the current CU. First, the coefficients can be derived from adjacent reconstructed pixels, and second, they can also be inherited from previously decoded blocks.

[0071] Derivation of EIP coefficients.

[0072] The decoder decodes the relevant syntax elements to determine the reconstructed region and filter shape of the selected type for the current block. For example... Figure 7 As shown, the selected filter moves horizontally or vertically within the selected reconstruction region in a one-pixel step to construct the autocorrelation matrix and cross-correlation vector. Specifically, three types of reconstruction regions are defined. The size of the reconstruction region depends on min(block width, block height) and the shape of the selected filter. For example, when the current block is... The block and the selected filter shape is At that time, the upper dimension of the reconstructed region is equal to... And the left dimension of the reconstructed region is equal to .

[0073] The coefficients calculated from the autocorrelation matrix and cross-correlation vector are the same as those in CCCM (Convolutional Cross Component Model).

[0074] Inheritance of EIP filters.

[0075] The EIP merging mode is also used in JVET-AF0080. Filter shapes and coefficients can be inherited from previously decoded blocks using EIPs or EIP merging modes. The decoder decodes the EIP merging flag to determine whether to use the proposed merging mode when the current block uses an EIP mode. When the EIP merging flag is true, the merging index is further decoded. The EIP merging list includes spatially neighboring and non-nearest candidates, temporal candidates, and historical candidates. The constructed EIP merging list can include up to 12 candidates, and this can be reduced to up to 6 candidates through a reordering process based on the SAD cost measured on an L-shaped template with a column width and row height of 1. In SAD computation, predictions for template regions via EIP filters are generated only from the reconstructed (neighboring and template) samples, allowing EIP filters to be applied in parallel rather than sequentially.

[0076] The spatial proximity, temporal, non-proximity, shift time, and historical candidate positions and inclusion order are the same as those defined in ECM-10.0 for merging prediction candidates for Cross Component Prediction (CCP).

[0077] Prediction for the current block.

[0078] The EIP mode generates the predicted value of the current block by predicting diagonally from the top left to the bottom right position, such as... Figure 8 As shown.

[0079] The predicted values ​​in JVET-AF0080 are calculated as follows: , in, In the current block The predicted value at that location, It is the i-th coefficient of the selected EIP filter, and the coefficient indices range from 0 to 14. It is a reconstruction or prediction value used for the current location, and and These are the position offsets relative to the current position along the x and y directions, respectively.

[0080] Mapped to the LFNST / NSPT / MTS collection.

[0081] The JVET-AF0080 method uses the DIMD procedure to derive the intra-prediction mode of the current block based on samples predicted by EIP. Specifically, the horizontal and vertical gradients are computed for each predicted sample to construct an Oriented Gradient Histogram (HoG). The intra-prediction mode corresponding to the maximum histogram count is then used to determine the LFNST, NSPT, or MTS transform set.

[0082] The proposed CU-level syntax.

[0083] At the CU level, signals are used to notify the syntax related to EIP. The following shows an example of the syntax related to EIP.

[0084] Figure 9The illustration depicts a method for improving prediction performance by mixing conventional predictions (non-EIP, e.g., directional, planar, DC) with EIP predictions, according to one embodiment. This is accomplished by employing N conventional intra-frame predictors along with EIP predictions. Template analysis can be performed to select the modes to be mixed and their corresponding weights. Hereinafter, EIPs as described in JVET-AF0080 are used. However, this method can be applied to more general EIPs, where, for example, filter size, filter coefficients, and / or EIP modes may differ from those described in JVET-AF0080.

[0085] In one implementation, we specifically propose using DIMD and TIMD template analysis. That is, we propose the following combination as an implementation of the general idea of ​​combining EIP and intra-prediction modes: - EIP and DIMD - EIP and TIMD.

[0086] A combination of EIP and DIMD.

[0087] As mentioned earlier, if DIMD fusion is applied, DIMD prediction is performed by combining planar prediction and five-angle prediction. To adopt EIP, the following is proposed.

[0088] Method 1: Conditionally replace the planar pattern In this method, instead of mixing the planar pattern with the angular pattern obtained through template analysis via DIMD, EIP is used. That is, EIP can be mixed with the angular pattern because EIP is considered a non-angular pattern like DC or the planar pattern.

[0089] Planar patterns can be continuously replaced by EIPs. Alternatively, template analysis can be used to determine whether to use an EIP or a planar pattern. That is, both EIPs and planar patterns are applied to the reconstructed template, and the template cost (typically SATD) is used to compare the two predictions. The prediction with the lower SATD cost is selected and mixed with other angular predictions.

[0090] The EIP mode considered here can be one of its three modes (full, horizontal, and vertical). Furthermore, all EIP modes from the merged candidates (local, non-local, historical, and time-based candidates) can be used. Template analysis is employed to select the optimal mode to replace the flat mode at both the encoder and decoder (no explicit signaling required).

[0091] Method 2: Conditionally replace other angle modes Instead of replacing the planar pattern, other angular patterns can also be replaced. This is also accomplished by using template analysis to select the optimal pattern. That is, EIP and other patterns derived from the DIMD process are applied to the reconstructed template, and the template cost is used to compare predictions. The predictor with the lowest cost (including EIP and other conventional patterns) is selected and mixed according to the DIMD mixing process.

[0092] The EIP pattern considered here can be one of its three modes (full, horizontal, and vertical). Furthermore, all EIP patterns from the merge candidates (local, non-local, historical, and time-based candidates) can be used. Template analysis is employed to select the optimal EIP pattern to replace the conventional pattern.

[0093] Method 3: Select the EIP mode for mixing based on the positional correlation of DIMD.

[0094] For a given block using DIMD, if one of the histogram magnitude values ​​above and to the left is greater than the other... Doubled, for example This allows modification of the weights of each of the five export modes. As another example, .

[0095] If the histogram value above is the same as the histogram value on the left... If the value is increased, the weight can be represented by a vertical relationship, that is, it depends on the coordinate y of the current pixel in the block to be predicted. In this case, a vertical EIP mode can be systematically selected for blending among EIP full mode, EIP horizontal mode, and EIP vertical mode. If the amplitude of the left histogram is greater than the amplitude of the upper histogram... If the weight is increased, the weight can be characterized by a horizontal relationship, i.e., it depends on the coordinate x of the current pixel in the block to be predicted. In this case, a horizontal EIP mode can be systematically selected for blending from EIP full mode, EIP horizontal mode, and EIP vertical mode.

[0096] Method 4: Add DIMD-EIP as a new mode.

[0097] The above method modifies the DIMD prediction of the current block by replacing some intra-frame modes with EIP modes. To increase encoder flexibility, a new mode, named DIMD-EIP, is proposed to be signaled, indicating that DIMD is used in conjunction with EIP. If this flag is zero, regular DIMD is used. The following options were considered: Option 1: Signal this sub-mode that is DIMD. In this option, if DIMD is enabled for the current block, a flag is signaled to indicate whether EIP is combined with DIMD. In this case, the EIP modes of the current block and / or the merged block are evaluated on the template to determine which one to select and combine with other regular modes.

[0098] Option 2: Signal this mode as a sub-mode of EIP.

[0099] In this option, the flag is signaled to indicate whether EIP is combined with DIMD only if EIP is enabled for this block. Compared to the first option, this option requires less signaling and less template analysis. This is because the combination of DIMD and EIP is only allowed if the encoder has already selected EIP mode. That is, template analysis is not required to determine whether to use planar mode or EIP mode. In this case, the selected EIP mode will replace the planar mode.

[0100] The combination of EIP and TIMD.

[0101] TIMD derives two intra-modes by evaluating several intra-prediction modes on a reconstruction template and measuring the template cost, typically SATD. The two modes, associated with the two minimum costs, are then blended based on their template costs. For combination with EIP, consider the following approach.

[0102] Method 1: Include the EIP in the TIMD.

[0103] Instead of evaluating regular intra-frame modes separately, a method is proposed that also evaluate EIP modes on the reconstruction template. EIP modes correspond to the three EIP modes of the current block and / or merged candidate EIP modes. Template costs for all modes, including EIP modes, are compared to produce the two modes with the lowest costs. Based on template costs, these two modes can then be either regular or EIP modes. The two modes are blended using a template cost-based TIMD blending weight.

[0104] Another alternative approach is to modify the TIMD pattern so that non-angular patterns are always mixed with other angular patterns. The non-angular patterns are either planar / DC or EIP patterns. The optimal non-angular pattern is selected based on the template distance, and the remaining two candidates are derived using the standard TIMD procedure.

[0105] Method 2: Add TIMD-EIP as a new mode.

[0106] The method described above modifies the TIMD prediction of the current block by replacing some intra-frame modes with EIP modes. To increase encoder flexibility, a new mode (named TIMD-EIP) is proposed, indicating that TIMD is used in conjunction with EIP. If this flag is zero, regular TIMD is used.

[0107] The flag is signaled only if EIP is enabled for this block. Compared to the first option, this option requires less signaling and less template analysis. This is because the combination of TIMD and EIP is only allowed if the encoder has already selected EIP mode. That is, template analysis is not required to determine whether to use planar mode or EIP mode. In this case, the selected EIP mode will replace the planar mode.

[0108] EIP mode application.

[0109] In one approach, the application of the EIP filter is performed independently of other mixing modes (e.g., DIMD, TIMD). This means that intra-frame predictions are computed independently and then mixed together.

[0110] In another approach, an EIP filter is applied to the predicted mixture: - First, calculate the predictions excluding EIP.

[0111] Then, for each sample, following the EIP process, each sample is computed using the EIP filter and then mixed with other predictions. In this case, the samples used by the EIP process are the mixed samples.

[0112] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first," "second," etc., can be used in various embodiments to modify elements, components, steps, operations, etc., e.g., "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply a sequence of modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or in a time period overlapping with the second decoding.

[0113] The various methods and other aspects described in the application itself can be used to modify, for example, Figure 2 and Figure 3The illustrated video encoder 200 or decoder 300 modules include, for example, intra-frame prediction modules (260, 360). Furthermore, this aspect is not limited to ECM, VVC, or HEVC, and can be applied to, for example, other standards and recommendations, and any extensions to such standards and recommendations. Unless otherwise stated or technically excluded, the aspects described in this application may be used alone or in combination.

[0114] Various numerical values ​​are used in this application. Specific values ​​are used for example purposes, and the aspects described are not limited to these specific values.

[0115] Various implementations involve decoding. As used herein, “decoding” can encompass all or part of a process performed, for example, on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. It will be clear, and is considered well understood by those skilled in the art, whether the phrase “decoding process” is intended to specifically refer to a subset of operations or generally to a broader decoding process, based on the context of the specific description.

[0116] Various implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” “encoding,” as used in this application, can encompass all or part of the process performed, for example, on an input video sequence, in order to produce an encoded bitstream.

[0117] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), implementations of the discussed features can be implemented in other forms (e.g., apparatus or program). Apparatus can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in an apparatus, such as a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0118] The references to "an embodiment" or "an embodiment" or "an implementation" or "implementation," and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation," and any other variations appearing throughout this application, do not necessarily refer to the same embodiment.

[0119] Furthermore, this application may refer to "determining" each piece of information. Determining information may include, for example, one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0120] Furthermore, this application may refer to "accessing" individual pieces of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0121] Furthermore, this application may refer to "receiving" individual pieces of information. Like "access," "receiving" is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more of them. Moreover, "receiving" is generally referred to in one manner or another during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0122] To be understood, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many listed items as possible.

[0123] Furthermore, as used herein, the term "notify with signaling" specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder notifies the quantization matrix used for dequantization with a signal. In this way, in one embodiment, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters as well as other parameters, signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that notification with signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the term "notify with signaling" has been referred to above, the term "signal" can also be used as a noun herein.

[0124] It will be apparent to those skilled in the art that implementations can generate various signals that are formatted to carry, for example, information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

Claims

1. A method for video decoding, comprising: The first prediction of the image blocks is generated based on the intra-frame prediction mode based on the extrapolation filter. A second prediction for the block is generated based on another intra-frame prediction mode; The first prediction and the second prediction are combined to form a prediction for the block; as well as The block is decoded based on the prediction for the block.

2. A video encoding method, comprising: The first prediction of the image blocks is generated based on the intra-frame prediction mode based on the extrapolation filter. A second prediction for the block is generated based on another intra-frame prediction mode; The first prediction and the second prediction are combined to form a prediction for the block; as well as The block is encoded based on the prediction for the block.

3. An apparatus for video decoding, comprising at least one memory and one or more processors, wherein, The one or more processors are configured to: The first prediction of the image blocks is generated based on the intra-frame prediction mode based on the extrapolation filter. A second prediction for the block is generated based on another intra-frame prediction mode; The first prediction and the second prediction are combined to form a prediction for the block; as well as The block is decoded based on the prediction for the block.

4. An apparatus for video encoding, comprising at least one memory and one or more processors, wherein, The one or more processors are configured to: The first prediction of the image blocks is generated based on the intra-frame prediction mode based on the extrapolation filter. A second prediction for the block is generated based on another intra-frame prediction mode; The first prediction and the second prediction are combined to form a prediction for the block; as well as The block is encoded based on the prediction for the block.

5. The method according to claim 1 or 2, or the apparatus according to claim 3 or 4, wherein, The second prediction for the block is based on another intra-frame prediction mode based on an extrapolation filter.

6. The method according to claim 1 or 2, or the apparatus according to claim 3 or 4, wherein, The second prediction for the block is based on an intra-prediction mode, rather than an intra-prediction mode based on an extrapolation filter.

7. The method according to any one of claims 1, 2, 5 and 6, or the apparatus according to any one of claims 3-6, wherein, Templates including neighboring samples are used to determine which predictions to generate the first and second predictions.

8. The method according to claim 7 or the apparatus according to claim 7, wherein, The directional gradient histogram of the template used in the determination.

9. The method according to claim 8 or the apparatus according to claim 8, wherein, The other intra-frame prediction mode is an angular intra-frame prediction mode determined based on the orientation gradient histogram of the template.

10. The method according to claim 8 or 9, further comprising, or the apparatus according to any one of claims 8 and 9, wherein, The one or more processors are further configured to execute: Multiple angular intra-frame prediction patterns are obtained from the directional gradient histogram of the template; Obtain one or more intra-frame prediction modes based on extrapolation filters; Intra-prediction is performed on the template based on each mode in the intra-prediction mode set, the intra-prediction mode set including the plurality of angle intra-prediction modes and the one or more intra-prediction modes based on extrapolation filters. Based on each mode in the intra-prediction mode set, obtain the corresponding cost between the reconstructed sample of the template and the predicted sample of the template from the intra-prediction of the template; as well as A mode is selected from the set of intra-frame prediction modes based on the corresponding cost.

11. The method according to any one of claims 1, 2, and 5-10, or the apparatus according to any one of claims 3-10, wherein, The mixture is based on a weighted sum of the first prediction and the second prediction.

12. The method according to claim 11 or the apparatus according to claim 11, wherein, The weighting is based on the amplitude values ​​of the left and top histograms.

13. The method according to any one of claims 8-12 or the apparatus according to any one of claims 8-12, wherein, Use decoder-side intra-frame mode derivation (DIMD).

14. The method according to claim 13 or the apparatus according to claim 13, wherein, In response to DIMD being enabled, a signaling flag is used to indicate whether intra-frame prediction based on extrapolation filters will be combined with DIMD.

15. The method according to claim 13 or the apparatus according to claim 13, wherein, In response to EIP being enabled, a signaling flag is used to indicate whether intra-frame prediction based on extrapolation filters will be combined with DIMD.

16. The method according to claim 7 or the apparatus according to claim 7, wherein, The determination is performed by testing the extrapolation filter-based intra-prediction mode and the other intra-prediction mode in the template.

17. The method of claim 16, further comprising, or the apparatus of claim 16, wherein, The one or more processors are further configured to execute: Obtain multiple angle intra-prediction modes that differ from the intra-prediction modes based on extrapolation filters; Obtain one or more intra-frame prediction modes based on extrapolation filters; Intra-prediction is performed on the template based on each mode in the intra-prediction mode set, the intra-prediction mode set including the plurality of angle intra-prediction modes and the one or more intra-prediction modes based on extrapolation filters. Based on each mode of the intra-prediction mode set, obtain the corresponding cost between the reconstructed sample of the template and the predicted sample of the template from the intra-prediction of the template; as well as A mode is selected from the set of intra-frame prediction modes based on the corresponding cost.

18. The method according to any one of claims 1, 2, and 5-17, or the apparatus according to any one of claims 3-17, wherein, Template-based intra-frame mode derivation (TIMD) is used.

19. The method of claim 18 or the apparatus of claim 18, wherein, In response to TIMD being enabled, a signaling flag is used to indicate whether intra-frame prediction based on extrapolation filters is combined with TIMD.

20. The method of claim 18 or the apparatus of claim 18, wherein, In response to EIP being enabled, a signaling flag is used to indicate whether intra-frame prediction based on extrapolation filters will be combined with TIMD.

21. A signal comprising a bit stream, said bit stream being formed by performing the method of any one of claims 1, 2, and 5-20.

22. A computer-readable storage medium having stored thereon instructions for encoding or decoding video according to any one of claims 1, 2, and 5-20.

Images