Template-based filtering for inter prediction

Abstract

In one embodiment, prediction of a block is filtered using a filter learned on a template. In order to learn the filter, a first template is generated from a decoded sample set in a region adjacent to the current block, and a second template is generated from a prediction sample set in the same adjacent region, wherein the prediction sample set may be obtained based on prediction parameters for predicting the current block or from stored prediction samples of neighboring block (s) associated with the template region. Filter parameters are calculated by minimizing a loss function between the decoded sample set and the filtered prediction sample set. The filter may be a convolutional 7-tap filter, a convolutional 5-tap filter, or a piecewise linear function.

Description

Technical Field

[0001] The present embodiments generally relate to methods and apparatus for inter-frame prediction in video encoding and decoding. Background Art

[0002] In order to achieve high compression efficiency, image and video coding schemes usually use prediction and transformation to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame picture correlation, and then the difference between the original block and the predicted block (usually expressed as a prediction error or prediction residual) is transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transformation, and prediction. Summary of the invention

[0003] According to one embodiment, a video decoding method is proposed, which includes: obtaining one or more prediction parameters of a block to be decoded in a picture; obtaining a prediction block of the block based on the one or more prediction parameters; obtaining a decoded sample set in an area adjacent to the block; obtaining a prediction sample set in the area adjacent to the block; obtaining one or more filter parameters of a filter based on the decoded sample set and the prediction sample set in the area adjacent to the block; applying the filter to the prediction block of the block to form a filtered prediction block of the block; and decoding the block based on the filtered prediction block of the block.

[0004] According to another embodiment, a video encoding method is proposed, which includes: obtaining one or more prediction parameters of a block to be encoded in a picture; obtaining a prediction block of the block based on the one or more prediction parameters; obtaining a decoded sample set in an area adjacent to the block; obtaining a prediction sample set in the area adjacent to the block; obtaining one or more filter parameters of a filter based on the decoded sample set and the prediction sample set in the area adjacent to the block; applying the filter to the prediction block of the block to form a filtered prediction block of the block; and encoding the block based on the filtered prediction block of the block.

[0005] According to another embodiment, a device for video decoding is provided, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: obtain one or more prediction parameters of a block to be decoded in a picture; obtain a prediction block of the block based on the one or more prediction parameters; obtain a decoded sample set in an area adjacent to the block; obtain a prediction sample set in the area adjacent to the block; obtain one or more filter parameters of a filter based on the decoded sample set and the prediction sample set in the area adjacent to the block; apply the filter to the prediction block of the block to form a filtered prediction block of the block; and decode the block based on the filtered prediction block of the block.

[0006] According to another embodiment, a device for video encoding is provided, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: obtain one or more prediction parameters of a block to be encoded in a picture; obtain a prediction block of the block based on the one or more prediction parameters; obtain a reconstructed sample set in an area adjacent to the block; obtain a prediction sample set in the area adjacent to the block; obtain one or more filter parameters of a filter based on the reconstructed sample set and the prediction sample set in the area adjacent to the block; apply the filter to the prediction block of the block to form a filtered prediction block of the block; and encode the block based on the filtered prediction block of the block.

[0007] One or more embodiments also provide a computer program including instructions, which, when executed by one or more processors, cause the one or more processors to perform an encoding method or a decoding method according to any one of the embodiments described herein. One or more of the embodiments also provide a computer-readable storage medium having stored thereon instructions for performing video encoding or decoding according to the methods described herein.

[0008] One or more embodiments further provide a computer-readable storage medium on which video data generated according to the above method is stored. One or more embodiments further provide a method and apparatus for transmitting or receiving video data generated according to the method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 Illustrated is a block diagram of a system within which aspects of the present embodiments may be implemented.

[0010] Figure 2 A block diagram of an embodiment of a video encoder is illustrated.

[0011] Figure 3A block diagram of an embodiment of a video decoder is illustrated.

[0012] Figure 4 The illustration shows template matching performed on a search area around an initial MV.

[0013] Figure 5 Illustrated are the top and left neighboring blocks used in CIIP (Joint Inter Intra Prediction) weight derivation.

[0014] Fig. 6A and Figure 6B The diagram shows how the angular pattern is divided.

[0015] Figure 7 Illustrated is the input of the spatial 5-tap components to the filter when the CCCM (Convolutional Cross-Component Model) predicts the current chroma CB to be encoded / decoded from a potentially downsampled version of the reconstructed luma CB concatenated with the current chroma CB.

[0016] Fig. 8A illustrates a reconstructed luma CB juxtaposed with a current W×H chroma CB to be encoded / decoded, Figure 8B The downsampled reconstructed luminance CB and luminance reference region are illustrated, and Figure 8C Its chromaticity reference area is illustrated.

[0017] Fig. 9A and Fig. 9B The templates of decoded reference samples and the templates of predicted samples of the current W×H luma CB to be encoded / decoded are illustrated respectively.

[0018] Fig.10 Illustrated is template-based prediction filtering at the encoder side for a current block predicted in inter-mode according to one embodiment.

[0019] Fig.11 Illustrated is template-based prediction filtering at the decoder side for a current block predicted in inter-mode according to one embodiment. DETAILED DESCRIPTION

[0020] Figure 1A block diagram of an example of a system in which various aspects and embodiments can be implemented is illustrated. System 100 can be embodied as a device including various components described below, and is configured to perform one or more of the aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. The elements of system 100 can be embodied in a single integrated circuit, multiple ICs, and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 100 is configured to implement one or more of the aspects described in this application.

[0021] The system 100 includes at least one processor 110, which is configured to execute instructions loaded therein, for implementing various aspects described in the present application, for example. The processor 110 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a disk drive, and / or an optical drive. As a non-limiting example, the storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device.

[0022] The system 100 includes an encoder / decoder module 130, which is configured to process data to provide encoded video or decoded video, for example, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module (one or more) that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 130 may be implemented as an independent element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software as known to those skilled in the art.

[0023] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described in this application may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0024] In several embodiments, memory inside the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory outside the processing device (e.g., the processing device can be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory can be a memory 120 and / or a storage device 140, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations (such as for MPEG-2, HEVC, or VVC).

[0025] Input to the elements of system 100 may be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0026] In various embodiments, the input device of block 105 has associated corresponding input processing elements, as known in the art. For example, the RF part can be associated with elements suitable for the following: (i) selecting the desired frequency (also referred to as selecting a signal, or limiting the signal to a frequency band), (ii) down-converting the selected signal, (iii) again limiting the band to a narrower frequency band, to select the signal frequency band that can be called a channel (for example) in some embodiments, (iv) demodulating the down-converted and limited-band signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF part of various embodiments includes one or more elements for performing these functions, for example, a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF part can include a tuner that performs various functions in these functions, including, for example, down-converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or a baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and perform frequency selection by filtering, down-conversion and filtering to the desired frequency band again. Various embodiments rearrange the order of above-mentioned elements (and other elements), remove some in these elements, and / or add other elements that perform similar or different functions. Adding element can be included in inserting element between existing elements, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part comprises antenna.

[0027] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 100 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within the processor 110 as desired. Similarly, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within the processor 110 as desired. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and the encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on an output device.

[0028] The various components of system 100 may be disposed within an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data between using a suitable connection arrangement 115 (eg, an internal bus known in the art, including an I2C bus, wiring, and printed circuit boards).

[0029] The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented in, for example, a wired and / or wireless medium.

[0030] In various embodiments, data is streamed to the system 100 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals of these embodiments are received through a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 100 using a set-top box that passes data through an HDMI connection of an input block 105. Still other embodiments provide streaming data to the system 100 using an RF connection of an input block 105.

[0031] The system 100 can provide output signals to various output devices, including a display 165, a speaker 175, and other peripherals 185. In various examples of embodiments, other peripherals 185 include one or more of independent DVRs, disk players, stereo systems, lighting systems, and other devices that provide functions based on the output of the system 100. In various embodiments, control signals are transmitted between the system 100 and the display 165, the speaker 175, or other peripherals 185 using signaling such as AV Link, CEC, or other communication protocols to achieve device-to-device control with or without user intervention. The output device can be communicatively coupled to the system 100 via a dedicated connection through the corresponding interfaces 160, 170, and 180. Alternatively, the output device can be connected to the system 100 using a communication channel 190 via the communication interface 150. The display 165 and the speaker 175 can be integrated into a single unit with other components of the system 100 in an electronic device (e.g., a television). In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.

[0032] Display 165 and speaker 175 may alternatively be separate from one or more other components, for example if the RF portion of input 105 is part of a stand-alone set-top box. In various embodiments where display 165 and speaker 175 are external components, output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0033] Figure 2An example video encoder 200 is illustrated, such as a VVC (Versatile Video Coding) encoder. Figure 2 It is also possible to illustrate an encoder that has improved the VVC standard or an encoder that uses a technique similar to VVC.

[0034] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "encoded" or "coded" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0035] Before being encoded, the video sequence may undergo a pre-encoding process (201), for example, applying a color transform to an input color picture (e.g., from RGB 4:4:4 to YCbCr4:2:0), or performing a remapping of input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0036] In encoder 200, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (202) and processed in units of, for example, CUs (coding units). Each unit is encoded using, for example, intra mode or inter mode. When a unit is encoded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of intra mode or inter mode to use to encode the unit and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (210) the predicted block from the original image block.

[0037] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients along with motion vectors and other syntax elements (such as picture partition information) are entropy encoded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is directly encoded without applying the transform or quantization process.

[0038] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) / ALF (adaptive loop filter) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0039] Figure 3 A block diagram of an example video decoder 300 is shown. In the decoder 300, the bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs the same Figure 2 The decoding stage is the reverse of the encoding stage described in . The encoder 200 also typically performs video decoding as part of encoding the video data.

[0040] In particular, the input to the decoder includes a video bitstream that may be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, prediction modes, motion vectors, and other encoding information. Picture partition information indicates how the picture is partitioned. Thus, the decoder may partition (335) the picture according to the decoded picture partition information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residuals. The decoded prediction residuals and the prediction blocks are combined (355) to reconstruct the image block. The prediction block may be obtained (370) from intra-frame prediction (360) or motion compensated prediction (i.e., inter-frame prediction) (375). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored at a reference picture buffer (380). Note that for a given picture, the contents of the reference picture buffer 380 on the decoder 300 side are identical to the contents of the reference picture buffer 280 on the encoder 200 side for the same picture.

[0041] The decoded picture may further undergo post-decoding processing (385), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding process (201). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0042] The present disclosure relates to inter-coding. We propose to apply a filtering process to the prediction signal to improve the prediction and thus reduce the residual information signaled to the decoder side. The filter coefficients are learned from reconstructed neighboring samples, where no additional signaling is required because the same process is performed at both the encoder and decoder sides. In the following, some details about inter-coding in Enhanced Compression Model (ECM) and Versatile Video Coding (VVC) are presented.

[0043] Local Illumination Compensation (LIC) in ECM

[0044] LIC is an inter-prediction technique for modeling the local illumination variation between the current block and its prediction block as a function of the local illumination variation between the current block template and the reference block template. The parameters of the function can be represented by scalars α and offset β, which form a linear equation (i.e., α*p[x]+β) to compensate for illumination changes, where p[x] is the reference sample pointed to by the MV (motion vector) at position x in the reference picture. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required for them, except for signaling the LIC flag for AMVP (Advanced Motion Vector Prediction) mode to indicate the use of LIC.

[0045] Local illumination compensation is used for uni-predicted inter CUs with the following modifications.

[0046] Both inter-frame neighbor samples and intra-frame neighbor samples can be used in LIC parameter derivation;

[0047] Disable LIC for blocks with less than 32 luma samples;

[0048] For both non-subblock and affine modes, LIC parameter derivation is performed based on the template block samples corresponding to the current CU instead of the partial template block samples corresponding to the first top-left 16×16 unit;

[0049] Generate samples of the reference block template by using MC (motion compensation) with block MV (motion vector) without rounding them to integer pixel precision.

[0050] Template Matching(TM) in ECM

[0051] Template matching (TM) is a decoder-side MV derivation method to refine the motion information of the current CU by finding the closest match between the template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and the blocks in the reference picture (i.e., the same size as the template). Figure 4As shown in , within a pixel search range (e.g., [-8, +8] pixel search range), a better MV is searched around the initial motion of the current CU. In ECM, a template matching method is used, in which the search step size is determined based on the AMVR (Adaptive Motion Vector Resolution) mode, and the TM can be cascaded with the bilateral matching process in the merge mode.

[0052] In AMVP mode, MVP (motion vector prediction) candidates are determined based on template matching error, where the candidate with the minimum difference between the current block template and the reference block template is selected. TM is then performed only on this specific MVP candidate for MV refinement. TM refines the MVP candidate by using an iterative diamond search, starting from full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) within the [-8, +8] pixel search range. The AMVP candidate can be further refined by using a cross search with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) followed by half-pixel and quarter-pixel MVD accuracy, depending on the AMVR mode as specified in Table 1. This search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. During the search process, if the difference between the previous minimum cost and the current minimum cost in the iteration is less than a threshold (which is equal to the block area), the search process terminates.

[0053] Table 1. Search patterns of AMVR and merge patterns using AMVR

[0054]

[0055] In merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As shown in Table 1, depending on whether an alternative interpolation filter is used according to the merged motion information (which is used when AMVR is in half-pixel mode), TM can be performed all the way to 1 / 8 pixel MVD accuracy or skip those exceeding half-pixel MVD accuracy. In addition, when TM mode is enabled, template matching can work as an independent process between block-based and sub-block-based bidirectional matching (BM) methods or an additional MV refinement process, depending on whether BM can be enabled according to its enabling condition.

[0056] Joint Inter-Intra Prediction (CIIP) in VVC and ECM

[0057] In VVC, when a CU is encoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is signaled to indicate whether the joint inter / intra prediction (CIIP) mode is applied to the current CU. As the name implies, CIIP prediction combines the inter prediction signal with the intra prediction signal. The inter prediction signal P in CIIP mode is derived using the same inter prediction process applied to the regular merge mode. inter , and derives the intra prediction signal P after a conventional intra prediction process using planar mode intra Then, the intra prediction signal and the inter prediction signal are combined using a weighted average, where the weights of the top and left neighboring blocks ( Figure 5 The weight values ​​are calculated using the encoding mode depicted in the figure as follows:

[0058] - if the top neighbor is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0;

[0059] - if the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0;

[0060] -If (isIntraLeft+isIntraTop) is equal to 2, set wt to 3;

[0061] - Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, set wt to 2;

[0062] - Otherwise, set wt to 1.

[0063] The CIIP forecast is formed as follows:

[0064] P CIIP =((4-wt)*P inter +wt*P intra +2)>>2.

[0065] In ECM, CIIP is extended in the following way. The prediction samples are generated by weighting the inter prediction signal using CIIP-TM merge candidate prediction and the intra prediction signal predicted by the intra prediction mode derived using TIMD (Template-based Intra Mode Derivation). This method is only applicable to coding blocks with an area less than or equal to 1024.

[0066] The TIMD derivation method is used to derive intra prediction modes in CIIP. Specifically, the intra prediction mode with the smallest SATD (Sum of Absolute Transform Difference) value in the TIMD mode list is selected and mapped to one of the 67 conventional intra prediction modes.

[0067] In addition, it is proposed to modify the weights (wIntra, wInter) if the derived intra prediction mode is an angular mode. For near-horizontal modes (2≤angular mode index<34), the current block is split vertically, such as Fig. 6A As shown in; for near vertical mode (34≤angle mode index≤66), the current block is divided horizontally, such as Figure 6B as shown in .

[0068] (wIntra, wInter) for different sub-blocks are shown in Table 2.

[0069] Table 2. Modification weights for corner mode

[0070] Sub-block index (wIntra,wInter) 0 (6,2) 1 (5,3) 2 (3,5) 3 (2,6)

[0071] Using CIIP-TM, a CIIP-TM merge candidate list is built for the CIIP-TM mode. The merge candidates are refined by template matching. The CIIP-TM merge candidates are also reordered as regular merge candidates by the ARMC (Adaptive Reordering of Merge Candidates) method. Basically, when ARMC is used, merge candidates with a smaller template distance to the current block template are placed at the top of the list. The maximum number of CIIP-TM merge candidates is equal to two.

[0072] Convolutional Cross-Component Model (CCCM) in ECM

[0073] In the Exploration Experiments (EE) on top of ECM-4.0, the Convolutional Cross Component Model (CCCM) predicts the current chroma CB to be encoded / decoded by applying a convolution filter to a potentially downsampled version of the reconstructed luma CB collocated with the current chroma CB. When chroma subsampling is used, this downsampling is performed so that the resolution of the downsampled collocated reconstructed luma CB matches the resolution of the chroma grid.

[0074] The CCCM convolutional 7-tap filter consists of a 5-tap plus-shaped spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter consists of the center (C) luma sample concatenated with the current chroma sample to be predicted and its upper / north (N), lower / south (S), left / west (W), and right / east (E) neighbors, as Figure 7 as shown in .

[0075] The nonlinear term P is expressed as a power of two of the center brightness sample C and is scaled to the sample value range of the content.

[0076] P=(C*C+midVal)>>bitDepth

[0077] Where bitDepth represents the pixel bit depth, and midVal represents the middle value of the bit depth range. For example, for 10-bit content, it is calculated as

[0078] P=(C*C+512)>>10.

[0079] The bias term B represents the scalar offset between the input and output. B is set to the mid-range chrominance value, e.g., 512 for 10-bit content. 0 、c 1 、c 2 、c 3 、c 4 、c 5 and c 6 Using the seven coefficients of a 7-tap filter, the current predicted chroma sample "predChromaVal" is expressed as:

[0080] predChromaVal = clip(c 0 C+c 1 N+c 2 S+c 3 E+c 4 W+c 5 P+c 6 B)

[0081] where "clip" is clipped to the range of valid chroma sample values.

[0082] like Figure 8B and Figure 8C As shown in , the filter coefficients c are calculated by minimizing the mean square error (MSE) between the predicted chroma samples generated by applying a convolutional 7-tap filter to a potentially downsampled version of the reconstructed luma samples in the luma reference region (802) and the reconstructed chroma samples in the chroma reference region (803). 0 、c 1 、c 2 、c 3 、c 4 、c 5 and c 6 .

[0083] Fig. 8A A reconstructed luma CB is shown concatenated with a current WxH chroma CB (801) to be encoded / decoded. Figure 8BIllustrated is a downsampled reconstructed luma CB (800) juxtaposed with the chroma CB and a luma reference area (802) in the case where the chroma format is 4:2:0, ie, the resolution of each chroma channel is divided by 2 via subsampling prior to encoding. Figure 8C Its chrominance reference region (803) is illustrated.

[0084] The luma reference region (802) consists of six rows / columns of potentially downsampled reconstructed luma samples above and to the left of a potentially downsampled version of the reconstructed luma CB (800) juxtaposed with the current chroma CB. The chroma reference region (803) consists of six rows / columns of reconstructed chroma samples above and to the left of the current chroma CB (801) to be encoded / decoded. Each reference region is extended to the right by a CB width and by a CB height below the CB boundary. Each reference region is adjusted to include only available decoded reference samples. The regions need to be extended to support the edge samples of the plus-shaped spatial filter and to be padded when in unavailable regions.

[0085] MSE minimization is performed by calculating the autocorrelation matrix of the luminance input and the cross-correlation vector between the luminance input and the chrominance output. The autocorrelation matrix is ​​decomposed by LDL, and the final filter coefficients are calculated using back substitution (matrix inversion). This process follows the calculation of the adaptive linear filtering (ALF) filter coefficients in ECM, except that LDL decomposition is selected instead of Cholesky decomposition to avoid using square root operations. This calculation uses only integer arithmetic.

[0086] Note that a single model or a multi-model variant of CCCM can be used. The multi-model variant uses two models, one derived for samples above the average luma reference value and the other derived for the remaining samples. The multi-model CCCM mode can be selected for coding units (CUs) containing at least 128 available decoded reference samples.

[0087] Note also that the term "reference region" has been chosen to match the standard nomenclature of CCCMs. However, the reference region of a given CB is equivalent to the template of that CB.

[0088] In many ECM tools, the improvement in compression compared to VVC comes from the idea of ​​implicitly selecting the coding mode, for example by performing a RD (rate distortion) search both at the encoder and decoder side. The advantage of this approach is reduced signaling overhead, where no syntax elements are needed to describe the results of the RD search. For example, in the TM process, where a motion vector is refined around the current value using template comparison and selecting the motion vector that minimizes the difference between the template prediction and the reconstructed template. No signaling is required because the decoder performs the same operation.

[0089] In this paper, it is proposed to apply a filter to improve the inter prediction quality. The filter is learned from neighboring reconstructed samples, where no syntax elements are needed to describe the filter at the decoder side. The filtering operation can be considered as an improvement over the LIC process, where LIC can be seen as considering only Figure 7 The proposed new filter can consider the current center pixel and its surrounding pixels, similar to CCCM. Filtering for luma inter-frame prediction learned on template

[0090] For a given W×H luma CB to be encoded / decoded and predicted via a given inter prediction mode, the proposed filtering of the prediction of this CB via this inter prediction mode is decomposed into two steps: learning of the filter and applying the learned filter to the prediction of this CB. It should be noted that the proposed filtering method for inter prediction can also be applied to chroma components.

[0091] Learning Filters

[0092] Fig. 9A and Fig. 9B A template (902) of a prediction sample and a template (301) of a decoded reference sample of a current WxH luma CB (900) to be encoded / decoded are illustrated.

[0093] Fig. 9A The current W×H brightness CB(900) in Fig. 9B The first template (901) in the example consists of n pixels located above the current brightness CB. a The row reconstructs the reference sample and the n pixels to the left of the current brightness CB. l The current brightness CB also has Fig. 9B The second template (902) in the example consists of n pixels located above the current brightness CB. a The row prediction sample and the nth row to the left of the current brightness CB l The prediction samples are constructed using the same parameters as used to construct the current CB prediction. Non-limiting examples of prediction parameters for constructing prediction samples include MV, merge index, reference picture information, affine parameters, AMVR parameters. To save computation, the prediction samples can be actual prediction samples stored during reconstruction / decoding of the template region (which are generated using prediction parameters of neighboring blocks).

[0094] Given the encoding / decoding partition history leading to the encoding / decoding of the current luma CB, the first template (901) of decoded reference samples can be adjusted such that unavailable reconstructed reference samples are excluded from (901). The second template (902) of prediction samples can be adjusted in the same way, i.e., unavailable prediction samples are excluded from (902). For example, in Fig. 9AIn the example, the template (901) is adjusted to exclude the unavailable decoded reference samples at its bottom. b ∈[0,H] row (904) and the n unavailable reconstructed reference samples on its right hand side r ∈[0,W] columns (903). Similarly, the template (902) is adjusted to exclude n unavailable prediction samples at its bottom b ∈[0,H] row (904) and the n unavailable prediction samples on its right hand side r ∈[0,W] columns (903). Typically, the first template of decoded reference samples (901) and the second template of predicted samples (902) cover the same area in the picture.

[0095] Then, different possible filter parameters are tested to select the filter parameter θ. In particular, the possible filter parameters are applied to the second template of the prediction sample. The difference (e.g., MSE) between the filtered prediction sample and the decoded reference sample is calculated. The filter parameter θ can be learned by minimizing the MSE between the filtered luminance samples in the template (902) of the prediction sample and the reconstructed reference sample in the template (901) of the reconstructed reference sample. If necessary, the template (901) of the reconstructed reference sample and the template (902) of the prediction sample can be padded in the same manner using a p-pixel padding boundary. The padding can consist of filling the padding area with a given value. Alternatively, the padding can consist of copying the values ​​of available spatially adjacent samples to the given sample to be padded.

[0096] Apply the filter to the prediction of the current brightness CB

[0097] The filter with learned parameters θ can be applied to the inter-prediction blocks. Fig.10 Illustrated is template-based prediction filtering at the encoder side for a current block predicted in inter-mode according to one embodiment.

[0098] exist Fig.10 In the figure, the dotted line indicates that on the encoder side, the learning step must be carried out before running the filtering of the prediction of the current luma CB, but not necessarily just before that. In practice, several processes can be placed between the learning step and filtering the prediction of the current luma CB. For example, the learning step can be done once the template of the decoded reference samples has been reconstructed for the current luma CB.

[0099] In particular, at step 1010, the encoder extracts a template of a decoded reference sample from an adjacent reconstruction region of the current block, and the encoder also obtains a template of a prediction sample of the current block. At step 1020, filter parameters θ are learned based on the template of the decoded reference sample and the template of the prediction sample.

[0100] As described above, the prediction samples in the second template can be generated by using the prediction parameters of the current block. Alternatively, in order to save calculations, the prediction samples can be extracted from the template area, that is, the prediction samples for the neighboring blocks (based on the prediction parameters used to encode / decode the neighboring blocks) are stored and can be directly extracted for use in the second template.

[0101] Then, the prediction of the current block can be Perform (1030) filtering of the current brightness CB, thereby producing a filtered prediction Calculation (1040) in filter prediction The difference between and the original block (X) is used to obtain the prediction residual. Then, as Figure 2 As shown in , the residual can be quantized, transformed and entropy encoded.

[0102] Fig.11 Illustrated is template-based prediction filtering at the decoder side for a current luma CB predicted in inter mode according to one embodiment.

[0103] Similar to the encoder side, Fig.11 In , the dotted line indicates that on the decoder side, the learning step must be carried out before starting the filtering of the prediction of the current brightness CB, but not necessarily right before that.

[0104] In particular, at step 1110, the decoder extracts a template of a decoded reference sample from an adjacent decoded region of the current block, and the decoder also obtains a template of a prediction sample of the current block. The template is generated in the same manner as at the encoder side. At step 1120, based on the template of the decoded reference sample and the template of the prediction sample, the filter parameter θ (1120) is learned.

[0105] Then, the prediction of the current block can be Perform (1130) filtering of the current brightness CB, thereby producing a filtered prediction The reconstructed residual of the current brightness CB from the inverse transform With filtering prediction Combine (1140) to form the reconstructed current brightness like Figure 3 As shown in , the reconstructed blocks can be further filtered.

[0106] As a filter like convolutional 7-tap filter in CCCM

[0107] The filter can be a convolutional 7-tap filter as in CCCM. In this case, the input to the spatial 5-tap component of the filter can consist of the center (C) luma prediction sample and its upper / north (N), lower / south (S), left / west (W), and right / east (E) neighbors. Moreover, θ = {c 0, c 1 , c 2 , c 3 , c 4 , c 5 , c 6 The definitions of nonlinear terms and bias terms may follow those described for CCCM. Any other definitions of nonlinear terms or bias terms may also apply.

[0108] Filter as a convolutional 5-tap filter

[0109] The filter can be a convolutional 5-tap filter. This case can be equivalent to the case proposed for CCCM, but the nonlinear term and the bias term are removed. θ = {c 0 , c 1 , c 2 , c 3 , c 4}.

[0110] Filter as a piecewise linear function

[0111] The filter can be The piecewise linear function f of the segment i. Note that since the piecewise linear function can be expressed in different forms, its parameter set can take different forms. For example, a piecewise linear function can be expressed by giving the linear segment with index i the slope α i and offset β i and predefine its boundaries (b i , b i+1 ). In this case, θ={(α i , β i )} i∈[0，ρ-1] For example, if ρ = 4, b 0 =0, b 1 =120, b 2 =570, b 3 =1003, b 4 = 1024, then for a given input predicted brightness sample and its output filtered version

[0112]

[0113] In this example, the choice of filter parameter θ is the sample value to be filtered function of , but it could be, for example, a locally pre-filtered value (which could be useful for denoising before applying f()) or in response to local gradient activity as proposed in JVET-Z0140 (see Che-Wei Kuo et al., “AHG12: Enhanced CCLM”, JVET-Z0140, 26th JVET Meeting, by teleconference, 20–29 April 2022).

[0114] Interaction with LIC

[0115] As mentioned before, LIC can be considered as a special case of the proposed idea, where the filter is a single tap filter. That is, the center pixel is filtered (multiplied) and a shift is added.

[0116] In one embodiment, the LIC may be replaced by the proposed filtering operation. The advantage of this approach is that no additional flag is required to indicate that the filter is being used, since the LIC flag is reused to indicate that a filtering operation is performed.

[0117] Interaction with TM

[0118] In TM, the best motion is found by minimizing the distance (e.g., SSD, SAD, SATD) between the predicted block template and the current block template. The TM mode can be further improved by learning the optimal filter parameters after the template matching operation. That is, for the TM mode, the filtering operation is performed continuously. This leads to a new mode, called template matching plus filtering (TM+filtering), which does not require further signaling because the TM flag is reused to indicate TM+filtering.

[0119] CIIP and GPM interaction

[0120] In CCIP, a joint prediction is generated from both the inter part and the intra part. This joint prediction requires specific considerations for calculating the filter coefficients. This is also true for the geometric prediction mode (GPM) when one part is predicted by intra and the other by inter. The following options are considered:

[0121] 1. Learn a single filter for the inter part without filtering the intra prediction.

[0122] 2. Learn two separate filters for the inter-frame part and the intra-frame part to filter the two prediction signals.

[0123] 3. Disable the filtering process for these modes.

[0124] Two-way and multiple hypothesis interactions

[0125] Similar to the previous embodiments, when bi-prediction or multiple hypotheses are used, a combination of prediction signals is performed. The following options are considered:

[0126] 1. Learn multiple separate filters for each prediction signal and filter both prediction signals with separate filters.

[0127] 2. Apply a single filter to the final prediction signal.

[0128] 3. Disable the filtering process for these modes.

[0129] In the above, we mainly describe our proposed method for coding units or coding blocks. It should be noted that the proposed method can be applied at other partition levels or block levels using inter-frame prediction.

[0130] Filter parameter calculation for merge mode

[0131] In the case of merge mode, instead of using the template of the current block and the reference block to calculate the filter parameters, another approach is proposed. Basically, merge mode refers to another coded block predicted with the same motion information compared to the current block. Therefore, it can be assumed that the statistics of the prediction / reconstruction signals of the two blocks (current block and merged block) are similar. Then, it is more sensible to obtain the filter parameters that minimize the distance between the prediction signal and the reconstructed signal of the merged block (i.e., the merged block is the template region). This filter is then used to improve the prediction of the current block. It should be noted that this process is done both on the encoder side and on the decoder side, where the merged block is available on both sides.

[0132] In the case of sub-block transform (SBT), where the inter-predicted block is split into two sub-blocks with a predefined transform pair, the filtering operation can be modified as follows: the filter parameters are calculated from the first partition (i.e., the first partition is used as a template for the transmitted partition) and applied to the next partition. This is because both partitions have the same prediction and therefore it can be assumed that both partitions have similar statistics. It should be noted that this process is done both on the encoder side and on the decoder side, where merge blocks are available on both sides.

[0133] Various methods are described herein, and each method includes one or more steps or actions for realizing the method.Unless the correct operation of the method requires the specific order of steps or actions, otherwise the order and / or use of specific steps and / or actions can be modified or combined.Additionally, terms such as "first", "second" etc. can be used to modify elements, components, steps, operations, etc., such as "first decoding" and "second decoding" in various embodiments.Unless specifically required, the use of such terms does not mean the sequencing of the operation to modification.Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can occur in a time period overlapping with the second decoding, such as before the second decoding, during the second decoding, or with the second decoding.

[0134] The various methods and other aspects described in this application can be used to modify modules, such as Figure 2 and Figure 3 2 and 3. The inter-frame prediction module (270, 375) of the video encoder 200 and decoder 300 shown in FIG. 2 is a block diagram of a decoder 300. In addition, the present aspects are not limited to ECM, VVC or HEVC, and can be applied to, for example, other standards and recommendations, as well as extensions of any such standards and recommendations. Unless otherwise indicated or technically excluded, the aspects described in this application can be used alone or in combination.

[0135] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0136] Various implementations involve decoding. "Decoding" as used in this application may encompass, for example, all or part of a process performed on a received coded sequence in order to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. Based on the context of the specific description, whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to a broader decoding process will be clear and is considered to be well understood by those skilled in the art.

[0137] Various embodiments relate to encoding.In a manner similar to the discussion above regarding "decoding", "encoding" as used in this application may encompass all or part of the processes performed on an input video sequence, for example, to produce an encoded bitstream.

[0138] Embodiments and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single embodiment form (e.g., discussed only as a method), the embodiments of the features discussed can also be implemented in other forms (e.g., devices or programs). Devices can be implemented in, for example, suitable hardware, software, and firmware. The method can be implemented in, for example, a device, such as a processor, which generally refers to a processing device (including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device). The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0139] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearance of the phrase "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0140] Additionally, the present application may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0141] Additionally, this application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0142] Additionally, the present application may refer to "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.

[0143] It should be appreciated that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," use of any of the following " / ," "and / or," and "at least one of..." is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many items as listed, as will be apparent to one of ordinary skill in this and related arts.

[0144] In addition, as used herein, the term "signaling" refers, among other things, to indicating something to a corresponding decoder. For example, in some embodiments, the encoder signals a quantization matrix for dequantization. In this way, in one embodiment, the same parameters are used on both the encoder side and the decoder side. Thus, for example, the encoder can transmit (explicitly signal) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit savings are achieved in various embodiments. It should be appreciated that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the term "signaling" is mentioned above, the term "signal" can also be used as a noun in this article.

[0145] As will be apparent to one of ordinary skill in the art, embodiments may generate various signals that are formatted to carry information that can be, for example, stored or transmitted. The information may include, for example, instructions for executing a method, or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

Claims

1. A video decoding method, comprising: Obtaining one or more prediction parameters for a block to be decoded in a picture; obtaining a prediction block for the block based on the one or more prediction parameters; Obtaining a set of decoded samples in a region adjacent to the block; Obtaining a set of prediction samples in the region adjacent to the block; obtaining one or more filter parameters of a filter based on the set of decoded samples and the set of predicted samples in the region adjacent to the block; applying the filter to the prediction block for the block to form a filtered prediction block for the block; and The block is decoded based on the filtered prediction block of the block.

2. A video encoding method, comprising: Obtaining one or more prediction parameters for a block to be encoded in the picture; obtaining a prediction block for the block based on the one or more prediction parameters; Obtaining a set of decoded samples in a region adjacent to the block; Obtaining a set of prediction samples in the region adjacent to the block; obtaining one or more filter parameters of a filter based on the set of decoded samples and the set of predicted samples in the region adjacent to the block; applying the filter to the prediction block for the block to form a filtered prediction block for the block; and The block is encoded based on the filtered prediction block of the block. 3 . The method according to claim 1 , wherein the set of prediction samples is obtained based on the one or more prediction parameters of the block.

4. The method according to claim 1 or 2, wherein the set of prediction samples is obtained based on prediction samples stored from decoding one or more neighboring blocks associated with the template region.

5. The method according to any one of claims 1-4, wherein the one or more filter parameters are obtained by minimizing a loss function between the set of decoded samples and a filtered version of the set of prediction samples in the region adjacent to the block.

6. The method according to any one of claims 1 to 5, wherein the decoded sample set comprises samples above the block and samples to the left of the block.

7. The method according to any one of claims 1-6, wherein the filter is a convolution filter.

8. The method of any one of claims 1-7, wherein the one or more filter parameters include a single weight and a single bias.

9. The method according to any one of claims 1 to 6, wherein the filter corresponds to a piecewise linear filter.

10. The method according to any one of claims 1-9, wherein the one or more prediction parameters include at least a motion vector, reference picture information, merge mode information, and AMVR (Adaptive Motion Vector Resolution) information.

11. The method according to any one of claims 1-10, wherein a flag is used to indicate that template matching is used for prediction, and wherein the flag also indicates that the filter is applied.

12. The method according to any one of claims 1 to 11, further comprising: Another prediction of the block is obtained, wherein the another prediction is filtered with another filter different from the filter.

13. The method according to any one of claims 1 to 11, further comprising: obtaining another prediction of the block; and The prediction and the another prediction for the block are combined to form a multi-hypothesis prediction for the block, wherein the multi-hypothesis prediction is filtered with the filter.

14. Method according to claim 12 or 13, wherein the further prediction of the block is obtained from intra prediction or from a geometric prediction mode.

15. The method according to any one of claims 1-14, wherein adjacent merged blocks are selected as template regions.

16. An apparatus for video decoding, comprising one or more processors and at least one memory, wherein the one or more processors are configured to: Obtaining one or more prediction parameters for a block to be decoded in a picture; obtaining a prediction block for the block based on the one or more prediction parameters; Obtaining a set of decoded samples in a region adjacent to the block; Obtaining a set of prediction samples in the region adjacent to the block; obtaining one or more filter parameters of a filter based on the set of decoded samples and the set of predicted samples in the region adjacent to the block; applying the filter to the prediction block for the block to form a filtered prediction block for the block; and The block is decoded based on the filtered prediction block of the block.

17. An apparatus for video encoding, comprising one or more processors and at least one memory, wherein the one or more processors are configured to: Obtaining one or more prediction parameters for a block to be encoded in the picture; obtaining a prediction block for the block based on the one or more prediction parameters; Obtaining a set of decoded samples in a region adjacent to the block; Obtaining a set of prediction samples in the region adjacent to the block; obtaining one or more filter parameters of a filter based on the set of decoded samples and the set of predicted samples in the region adjacent to the block; applying the filter to the prediction block for the block to form a filtered prediction block for the block; and The block is encoded based on the filtered prediction block of the block.

18. The apparatus according to claim 16 or 17, wherein the set of prediction samples is obtained based on the one or more prediction parameters of the block.

19. The apparatus of claim 16 or 17, wherein the set of prediction samples is obtained based on prediction samples stored from decoding one or more neighboring blocks associated with the template region.

20. The apparatus of any one of claims 16-19, wherein the one or more filter parameters are obtained by minimizing a loss function between the set of decoded samples and a filtered version of the set of prediction samples in the region adjacent to the block.

21. The apparatus of any one of claims 16-20, wherein the one or more processors are further configured to: Another prediction of the block is obtained, wherein the another prediction is filtered with another filter different from the filter.

22. The apparatus according to any one of claims 16 to 21, wherein adjacent merged blocks are selected as template regions.

23. A signal comprising video data, formed by performing a method according to any one of claims 2 to 15.

24. A computer-readable storage medium having stored thereon instructions for encoding or decoding a video according to the method according to any one of claims 1-15.

Images