Method and apparatus for adaptive motion compensation filtering

Abstract

Methods, apparatus, and non-transitory computer-readable storage media for video decoding and encoding are provided. In a video decoding method, a decoder may obtain a target motion vector of a current interframe coding block from a candidate list based on a plurality of first reconstruction sample points adjacent to the current interframe coding block, the candidate list including a plurality of motion vector candidates of the current interframe coding block; obtaining a plurality of first prediction sample points based on the target motion vector of the current inter-frame coding block; and in response to determining that adaptive motion compensation filtering is applied to the current inter-coded block, obtaining a plurality of filtered prediction samples based on the at least one template filter and the plurality of first prediction samples.

Description

[0001] Cross-references to related applications

[0002] This application is filed and claims priority to U.S. Provisional Application No. 63 / 457,371, filed April 5, 2023, entitled “Adaptive Motion Compensated Filtering for Bi-Prediction,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to video encoding / decoding and compression, and specifically, but not limited to, methods and apparatus for improving the encoding / decoding efficiency of inter-frame coded blocks. Background Technology

[0004] Various video codec techniques can be used to compress video data. Video codecs are performed according to one or more video codec standards. For example, video codec standards include Universal Video Codec (VVC), High Efficiency Video Codec (H.265 / HEVC), High-Advanced Video Codec (H.264 / AVC), and Moving Picture Experts Group (MPEG) codecs. Video codecs typically use predictive methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize redundancy present in video images or sequences. A key goal of video codec techniques is to compress video data into a form that uses a lower bitrate while avoiding or minimizing video quality degradation.

[0005] The first version of the VVC standard was completed in July 2020, offering approximately 50% bitrate savings or equivalent perceived quality compared to its predecessor, HEVC. While the VVC standard provides significant codec improvements over its predecessor, there is evidence that superior codec efficiency can be achieved using additional codec tools. Recently, the Joint Video Exploration Group (JVET), in collaboration with ITU-T VECG and ISO / IEC MPEG, began exploring advanced technologies that can significantly improve codec efficiency compared to VVC. In April 2021, a software codebase called the Enhanced Compression Model (ECM) was established for future video codec exploration work. The ECM reference software is based on the VVC Test Model (VTM) developed by JVET for VVC and further extends and / or improves several existing modules (e.g., intra / inter-frame prediction, transform, loop filters, etc.). In the future, any new codec tools beyond the VVC standard will need to be integrated into the ECM platform and tested using the JVET Common Test Conditions (CTC). Summary of the Invention

[0006] The present disclosure provides examples of techniques related to improving encoding / decoding efficiency of inter-coded blocks.

[0007] According to a first aspect of the present disclosure, a method of video decoding of an inter-coded block is provided. In the method, a decoder can obtain a target motion vector of a current inter-coded block from a candidate list based on a plurality of first reconstructed samples neighboring the current inter-coded block, wherein the candidate list comprises a plurality of motion vector candidates of the current inter-coded block; obtain a plurality of first predicted samples based on the target motion vector of the current inter-coded block; and in response to determining that an adaptive motion compensation filtering is applied to the current inter-coded block, obtain a plurality of filtered predicted samples based on at least one template filter and the plurality of first predicted samples, wherein the at least one template filter is obtained based on a current template of the current inter-coded block, wherein the current template comprises a plurality of second reconstructed samples neighboring the current inter-coded block.

[0008] According to a second aspect of the present disclosure, a method of video encoding of an inter-coded block is provided. In the method, an encoder can obtain a target motion vector of a current inter-coded block from a candidate list based on a plurality of first reconstructed samples neighboring the current inter-coded block, wherein the candidate list comprises a plurality of motion vector candidates of the current inter-coded block; obtain a plurality of first predicted samples based on the target motion vector of the current inter-coded block; and in response to determining that an adaptive motion compensation filtering is applied to the current inter-coded block, obtain a plurality of filtered predicted samples based on at least one template filter and the plurality of first predicted samples, wherein the at least one template filter is obtained based on a current template of the current inter-coded block, wherein the current template comprises a plurality of second reconstructed samples neighboring the current inter-coded block.

[0009] According to a third aspect of the present disclosure, an apparatus for video decoding is provided. The apparatus can include one or more processors and a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors. Furthermore, the one or more processors, upon execution of the instructions, are configured to perform the method according to the first aspect.

[0010] According to a fourth aspect of the present disclosure, an apparatus for video encoding is provided. The apparatus can include one or more processors and a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors. Furthermore, the one or more processors, upon execution of the instructions, are configured to perform the method according to the second aspect.

[0011] According to a fifth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium for storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform a method according to the first aspect.

[0012] According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium for storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform a method according to the second aspect.

[0013] According to a seventh aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium for storing a bitstream to be decoded by a method according to the first aspect.

[0014] According to an eighth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium for storing a bitstream generated by a method according to the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0015] More specific descriptions of examples of the present disclosure will be presented by referring to specific examples illustrated in the accompanying drawings. In view of the fact that these drawings only depict some examples and therefore should not be considered as limiting the scope, these examples will be described and explained in more detail by using the accompanying drawings.

[0016] FIG. 1A is a block diagram illustrating a system for encoding and decoding video blocks according to some examples of the present disclosure.

[0017] FIG. 1B is a block diagram of an encoder according to some examples of the present disclosure.

[0018] FIG. 1C to FIG. IF is a block diagram illustrating how frames are recursively partitioned into multiple video blocks having different sizes and shapes according to some examples of the present disclosure.

[0019] FIG. 1G is a block diagram illustrating an exemplary video encoder according to some examples of the present disclosure.

[0020] FIG. 2A is a block diagram of a decoder according to some examples of the present disclosure.

[0021] FIG. 2B is a block diagram illustrating an exemplary video decoder according to some examples of the present disclosure.

[0022] FIG. 3A is a diagram illustrating block partitioning in a multi-type tree structure according to some examples of the present disclosure.

[0023] FIG. 3B FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0024] FIG. 3C FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0025] FIG. 3D FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0026] FIG. 3E FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0027] FIG. 4 FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure. x and d y is one example of the horizontal and vertical values of the MV.

[0028] FIG. 5 FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0029] FIG. 6 shows an example of two diamond filter shapes, according to some examples of the present disclosure.

[0030] FIG. 7 FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0031] FIG. 8 FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0032] FIG. 9A and FIG. 9B FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0033] FIG. 10 FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0034] FIG. 11 FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0035] FIG. 12 FIG. 1 is a diagram illustrating block partitioning in a multi-type tree structure, according to some examples of the present disclosure.

[0036] FIG. 13 FIG. 1 is a diagram illustrating template-based unidirectional prediction sample adaptive motion compensation filtering, according to some examples of the present disclosure.

[0037] FIG. 14 FIG. 2 is a diagram illustrating an OBMC process for a CU coded without subblock motion compensation, according to some examples of the present disclosure.

[0038] FIG. 15 FIG. 3 is a diagram illustrating an OBMC process for a CU coded with subblock mode, according to some examples of the present disclosure.

[0039] FIG. 16 FIG. 4 is a diagram illustrating template-based OBMC, according to some examples of the present disclosure.

[0040] FIG. 17A and FIG. 17B FIG. 5 is a diagram illustrating non-adjacent neighboring blocks of different sizes, according to some examples of the present disclosure.

[0041] FIG. 18 FIG. 6 is a diagram illustrating a template used for cost calculation of non-subblock merge mode in ARMC and its corresponding reference samples, according to some examples of the present disclosure.

[0042] FIG. 19 FIG. 7 is a diagram illustrating a template used for cost calculation of subblock merge mode in ARMC and its corresponding reference samples, according to some examples of the present disclosure.

[0043] FIG. 20 FIG. 8 is a diagram illustrating refinement locations around one base candidate (reflected by the central star) along k x pi / 8 diagonal angles (reflected by the dots with three different patterns), according to some examples of the present disclosure.

[0044] FIG. 21 FIG. 9 is a diagram illustrating adaptive MC filtering applied when generating prediction samples of the current block but bypassed when calculating template cost of different merge / MMVD candidates, according to some examples of the present disclosure.

[0045] FIG. 22 FIG. 10 is a diagram illustrating adaptive MC filtering applied both when generating prediction samples of the current block and when calculating template cost of different merge / MMVD candidates, according to some examples of the present disclosure.

[0046] FIG. 23 FIG. 11 is a diagram illustrating a computing environment coupled with a user interface, according to some examples of the present disclosure.

[0047] FIG. 24is a flowchart illustrating a method of video decoding according to some examples of the present disclosure.

[0048] FIG. 25 is a flowchart illustrating a method of video encoding according to some examples of the present disclosure. DETAILED DESCRIPTION

[0049] Reference will now be made in detail to the Specific Embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth in order to assist in understanding the subject matter presented herein. But it will be apparent to one of ordinary skill in the art that the subject matter presented can be practiced without all of these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein can be implemented on many types of electronic devices with digital video capabilities.

[0050] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any one of the items in the list or any combination of items in the list.

[0051] Reference throughout this specification to “one embodiment,” “an embodiment,” “exemplary,” “some embodiments,” “some examples,” or similar language means that a particular feature, structure, or characteristic described follows in at least one embodiment or example. The particular feature, structure, element, or characteristic follows in at least one embodiment or example, unless specifically stated otherwise.

[0052] Throughout this disclosure the terms “first,” “second,” “third,” etc. are used merely as nomenclature to refer to relevant elements, such as devices, components, constituents, steps, etc., and do not necessarily imply any spatial or chronological order unless explicitly stated otherwise. For example, “a first device” and “a second device” can refer to two separately formed devices, or two portions, components, or states of operation of the same device, and can be named arbitrarily.

[0053] The terms "module," "sub-module," "circuit," "sub-circuit," "circuitry," "sub-circuitry," "unit," or "sub-unit" can include a memory (shared, dedicated, or group) that stores code or instructions that can be executed by one or more processors. A module can include one or more circuits with or without stored code or instructions. A module or circuit can include one or more components connected directly or indirectly to each other or to each other adjacent.

[0054] As used herein, the terms "if' or "when" can be understood to mean "upon" or "in response to," depending on the context. These terms, if present in a claim, can not indicate that the related limitation or feature is conditional or optional. For example, a method can include the steps of: i) performing function or action X' when or if condition X is present, and ii) performing function or action Y' when or if condition Y is present. The method can simultaneously possess the ability to perform function or action X' and the ability to perform function or action Y'. Thus, functions X' and Y' can be performed in different times in multiple executions of the method.

[0055] A unit or module can be implemented in pure software, in pure hardware, or in a combination of hardware and software. For example, in a pure software implementation, a unit or module can include functionally related code blocks or software components that are directly or indirectly linked together to perform a particular function.

[0056] FIG. 1A is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel, in accordance with some embodiments of the present disclosure. As shown in FIG. 1A As shown in FIG. 1, system 10 includes a source device 12 that generates and encodes video data to be decoded later by a destination device 14. Source device 12 and destination device 14 can comprise any of a wide variety of electronic devices including, for example, a cloud server, a server computer, a desktop or laptop computer, a tablet computer, a smart phone, a set-top box, a digital television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, etc. In some embodiments, source device 12 and destination device 14 are equipped with wireless communication capabilities.

[0057] In some embodiments, destination device 14 can receive, via link 16, encoded video data to be decoded. Link 16 can comprise any type of communication medium or device capable of moving the encoded video data from source device 12 to destination device 14. In one example, link 16 can comprise a communication medium to enable source device 12 to transmit encoded video data directly to destination device 14 in real-time. The encoded video data can be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium can comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. The communication medium can include routers, switches, base stations, or any other equipment that can be useful to facilitate communication from source device 12 to destination device 14.

[0058] In other embodiments, encoded video data can be transmitted from output interface 22 to storage device 32. Subsequently, encoded video data in storage device 32 can be accessed by destination device 14 via input interface 28. Storage device 32 can include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, Digital Video Disk (DVD), Compact Disk-Read Only Memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. In a further example, storage device 32 can correspond to a file server or another intermediate storage device that can hold the encoded video data generated by source device 12. Destination device 14 can access stored video data from storage device 32 via streaming or download. The file server can be any type of computer

[0059] As FIG. 1AAs shown in FIG. 1, source device 12 includes video source 18, video encoder 20, and output interface 22. Video source 18 can include a source such as a video capture device, e.g., a video camera, a video archive containing previously captured video, a video feed interface to receive video from a video content provider, and / or a computer graphics system for generating computer graphics data as the source video, or a combination of such sources. As one example, if video source 18 is a video camera of a security surveillance system, source device 12 and destination device 14 can form a camera phone or video phone. However, the implementations described in this application can be applied to video coding in general, and can have application to wireless and / or wired applications.

[0060] The captured, pre-captured, or computer-generated video can be encoded by video encoder 20. The encoded video data can be transmitted directly to destination device 14 via output interface 22 of source device 12. The encoded video data can also (or alternatively) be stored onto storage device 32 for later access by destination device 14 or other devices, for decoding and / or playback. Output interface 22 can further include a modem and / or a transmitter.

[0061] Destination device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 can include a receiver and / or modem and receives encoded video data over link 16. The encoded video data communicated over link 16, or provided on storage device 32, can include a variety of syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements can be included within the encoded video data transmitted on a communication medium, stored on a storage medium, or stored on a file server.

[0062] In some implementations, destination device 14 can include display device 34, which can be an integrated display device and an external display device configured to communicate with destination device 14. Display device 34 displays the decoded video data to a user, and can include any of a variety of display devices such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0063] Video encoder 20 and video decoder 30 can operate according to a proprietary standard or industry standard, such as VVC, HEVC, MPEG-4, Part 10, AVC, or extensions thereof. It should be understood that the application is not limited to a specific video coding / decoding standard and can be applicable to other video coding / decoding standards. It is generally contemplated that video encoder 20 of source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is also generally contemplated that video decoder 30 of destination device 14 can be configured to decode video data according to any of these current or future standards.

[0064] Video encoder 20 and video decoder 30 can be implemented as any of a variety of suitable encoder and / or decoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuitry, software, hardware, firmware or any combinations thereof. When implemented partially in software, an electronic device can store instructions for the software in a suitable, non- transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the video coding / decoding operations disclosed in the present disclosure. Each of video encoder 20 and video decoder 30 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined encoder / decoder (CODEC) in a respective device.

[0065] In some embodiments, at least a portion of the components of source device 12 (e.g., video source 18, video encoder 20, or the components included in video encoder 20 described below with reference to FIG. 2, and output interface 22) and / or at least a portion of the components of destination device 14 (e.g., input interface 28, video decoder 30, or the components included in video decoder 30 described below with reference to FIG. 3, and display device 34) can operate in a cloud computing service network, such as a software as a service (SaaS), platform as a service (PaaS), or infrastructure as a service (IaaS), which can provide software, platforms, and / or infrastructure. In some embodiments, one or more components of source device 12 and / or destination device 14 that are not included in the cloud computing service network can be provided in one or more client devices, and the one or more client devices can communicate with server computers in the cloud computing service network through a wireless communication network (e.g., a cellular communication network, a short-range wireless communication network, or a global navigation satellite system (GNSS) communication network) or a wired communication network (e.g., a local area network (LAN) communication network or a power-line communication (PLC) network). In embodiments, at least a portion of the operations described herein can be implemented as a cloud-based service provided by one or more server computers implemented by at least a portion of the components of source device 12 and / or at least a portion of the components of destination device 14 in the cloud computing service network; and one or more other operations described herein can be implemented by one or more client devices. In some embodiments, the cloud computing service network can be a private cloud, a public cloud, or a hybrid cloud. The terms such as “cloud,” “cloud computing,” “cloud-based,” and the like can be used interchangeably as appropriate, without departing from the scope of the disclosure. It should be understood that the disclosure is not limited to implementation in the above-described cloud computing service network. Rather, the disclosure can also be implemented in any other type of computing environment that is currently known or developed in the future.

[0066] Like HEVC, VVC is built on a block-based hybrid video coding framework. FIG. 1B is a block diagram illustrating a block-based video encoder according to some embodiments of the disclosure. In encoder 100, the input video signal is processed on a block-by-block basis, referred to as a coding unit (CU). Encoder 100 can be, for example, a video encoder included in a video encoding device, such as source device 12 in FIG. 1. In some embodiments, encoder 100 can be implemented in a cloud computing service network, such as a software as a service (SaaS), platform as a service (PaaS), or infrastructure as a service (IaaS), which can provide software, platforms, and / or infrastructure. In some embodiments, one or more components of encoder 100 that are not included in the cloud computing service network can be provided in one or more client devices, and the one or more client devices can communicate with server computers in the cloud computing service network through a wireless communication network (e.g., a cellular communication network, a short-range wireless communication network, or a global navigation satellite system (GNSS) communication network) or a wired communication network (e.g., a local area network (LAN) communication network or a power-line communication (PLC) network). In embodiments, at least a portion of the operations described herein can be implemented as a cloud-based service provided by one or more server computers implemented by at least a portion of the components of encoder 100; and one or more other operations described herein can be implemented by one or more client devices. In some embodiments, the cloud computing service network can be a private cloud, a public cloud, or a hybrid cloud. The terms such as “cloud,” “cloud computing,” “cloud-based,” and the like can be used interchangeably as appropriate, without departing from the scope of the disclosure. It should be understood that the disclosure is not limited to implementation in the above-described cloud computing service network. Rather, the disclosure can also be implemented in any other type of computing environment that is currently known or developed in the future. FIG. 1AThe video encoder 20 shown. In VTM-1.0, a CU can be up to 128x128 pixels. However, unlike HEVC which partitions a block based on quad-tree only, in VVC, a coding tree unit (CTU) is split into CUs to adapt to different local characteristics based on quad / bi / tri-tree. In addition, the concept of multiple partition unit types in HEVC is removed, i.e., there is no longer CU, prediction unit (PU) and transform unit (TU) partitioning in VVC; instead, each CU always serves as the basic unit for both prediction and transform without further partitioning. In the multi-type tree structure, a CTU is first partitioned by a quad-tree structure. Then, each quad-tree leaf node can be further partitioned by a binary tree structure and a ternary tree structure.

[0067] FIG. 3A to FIG. 3E is a schematic diagram illustrating multi-type tree partitioning modes according to some embodiments of the present disclosure. FIG. 3A to FIG. 3E Five partition types are shown, including quad partitioning FIG. 3A ), vertical binary partitioning FIG. 3B ), horizontal binary partitioning FIG. 3C ), vertical ternary partitioning FIG. 3D ), and horizontal ternary partitioning FIG. 3E .

[0068] For each given video block, spatial prediction and / or temporal prediction can be performed. Spatial prediction (or “intra prediction”) uses pixels from already coded neighboring blocks in the same video picture / slice (referred to as reference samples) to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in video signals. Temporal prediction (also referred to as “inter prediction” or “motion-compensated prediction”) uses reconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically represented by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference. Also, if multiple reference pictures are supported, a reference picture index is additionally signaled that is used to identify which reference picture from the reference picture store the temporal prediction signal comes from.

[0069] After spatial and / or temporal prediction, an intra / inter mode decision circuit 121 in the encoder 100 selects the best prediction mode, e.g., based on rate-distortion optimization methods. The block prediction value 120 is then subtracted from the current video block; and the resulting prediction residual is de-correlated using a transform circuit 102 and a quantization circuit 104. The resulting quantized residual coefficients are inverse quantized by an inverse quantization circuit 116 and inverse transformed by an inverse transform circuit 118 to form a reconstructed residual, which is then added back to the predicted block to form a reconstructed signal of the CU. Further, loop filters 115, such as a deblocking filter, a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF), can be applied to the reconstructed CU before it is placed in the reference picture store of the picture buffer 117 and used to code future video blocks. To form the output video bitstream 114, the coding mode (inter or intra), the prediction mode information, the motion information, and the quantized residual coefficients are all sent to an entropy encoding unit 106 for further compression and packing to form the bitstream.

[0070] For example, the deblocking filter is available in the current version of VVC as well as in AVC, HEVC. In HEVC, an additional loop filter called SAO is defined to further improve coding efficiency. In the current version of VVC standard, yet another loop filter called ALF is under active research and is very likely to be included in the final standard.

[0071] These loop filter operations are optional. Performing these operations helps to improve coding efficiency and visual quality. The encoder 100 can also decide to turn off these operations to save computational complexity.

[0072] It should be noted that if these filter options are turned on by the encoder 100, the intra prediction is usually based on unfiltered reconstructed pixels, while the inter prediction is based on filtered reconstructed pixels.

[0073] FIG. 2A is a block diagram illustrating a block-based video decoder 200 that can be used in conjunction with many video coding standards. The decoder 200 is similar to the encoder 100 located at FIG. 1B The decoder 200 is similar to the encoder 100 located at FIG. 1AThe illustrated video decoder 30. In the decoder 200, the incoming video bitstream 201 is first decoded by entropy decoding 202 to obtain quantized coefficient levels and prediction related information. The quantized coefficient levels are then processed by inverse quantization 204 and inverse transform 206 to obtain reconstructed prediction residuals. A block prediction mechanism implemented in intra / inter mode selector 212 is configured to perform intra prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by summing the reconstructed prediction residuals from inverse transform 206 with the prediction output generated by the block prediction mechanism using adder 214.

[0074] The reconstructed blocks can be further processed by loop filter 209 and then stored in picture buffer 213 for use as reference picture storage. The reconstructed video in picture buffer 213 can be sent out to drive a display device and also used for predicting future video blocks. In the case where loop filter 209 is turned on, filtering operations are performed on these reconstructed pixels to obtain the final reconstructed video output 222.

[0075] FIG. 1G is a block diagram illustrating another exemplary video encoder 20 according to some embodiments described in the present application. The video encoder 20 can perform intra prediction coding and inter prediction coding on video blocks within a video frame. Intra prediction coding relies on spatial prediction to reduce or remove spatial redundancy of video data within a given video frame or picture. Inter prediction coding relies on temporal prediction to reduce or remove temporal redundancy of video data within neighboring video frames or pictures of a video sequence. It should be noted that in the field of video coding, the term "frame" can be used as a synonym of the term "image" or "picture".

[0076] As FIG. 1GAs shown in FIG. 1, video encoder 20 includes video data memory 40, prediction processing unit 41, decoded picture buffer (DPB) 64, summer 50, transform processing unit 52, quantization unit 54, and entropy encoding unit 56. Prediction processing unit 41 further includes motion estimation unit 42, motion compensation unit 44, partition unit 45, intra-prediction processing unit 46, and intra block copy (BC) unit 48. In some implementations, video encoder 20 also includes inverse quantization unit 58, inverse transform processing unit 60, and summer 62 for video block reconstruction. A loop filter 63, such as a deblocking filter, can be located in the loop, after summer 62 and before DPB 64, to filter block boundaries to remove blockiness artifacts from reconstructed video. In addition to the deblocking filter, other loop filters (such as a sample adaptive offset (SAO) filter, a cross component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can be used to filter the output of summer 62. Note that for the CCSAO technique, the application is not limited to the embodiments described herein, but rather the application can be applied to the case where an offset is selected for any one of a luma component, a Cb chroma component, and a Cr chroma component to modify any other one of the luma component, the Cb chroma component, and the Cr chroma component based on the selected offset. Further, note that the first component referred to herein can be any one of the luma component, the Cb chroma component, and the Cr chroma component, the second component referred to herein can be any other one of the luma component, the Cb chroma component, and the Cr chroma component, and the third component referred to herein can be the remaining one of the luma component, the Cb chroma component, and the Cr chroma component. In some examples, the loop filters can be omitted, and the decoded video blocks can be directly provided to DPB 64 by summer 62. Video encoder 20 can take the form of fixed or programmable hardware implemented as one or more of the illustrated fixed or programmable hardware components.

[0077] Video data memory 40 can store video data to be encoded by the components of video encoder 20. The video data in video data memory 40 can be obtained, for example, from video source 18 as shown in FIG. 1. DPB 64 is a buffer that stores reference video data (e.g., reference frames or pictures) for use in encoding video data by video encoder 20 (e.g., in intra- or inter-coding modes). Video data memory 40 and DPB 64 can be formed by a variety of memory devices, including any combination of different memory devices. In various examples, video data memory 40 can be on-chip with other components of video encoder 20, or off-chip relative to those components. FIG. 1A

[0078] As FIG. 1G ​As shown in FIG. 1, after receiving video data, partitioning unit 45 within prediction processing unit 41 partitions the video data into video blocks. This partitioning can also include partitioning of a video frame into slices, tiles, or other larger coding units (CUs) according to a predefined splitting structure (e.g., a quadtree (QT) structure) associated with the video data, for example. A video frame is or can be considered as a two-dimensional array or matrix of sample values. Samples in the array can also be referred to as pixels or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the video frame. A video frame can be divided into a plurality of video blocks, for example, by using QT partitioning. A video block is or can be considered as a two-dimensional array or matrix of sample values again, but with dimensions smaller than those of the video frame. The number of samples in the horizontal and vertical directions (or axes) of the video block defines the size of the video block. A video block can be further partitioned into one or more block partitions or sub-blocks (which can form blocks again) by using QT partitioning, binary tree (BT) partitioning, or ternary tree (TT) partitioning, or any combination thereof, for example, iteratively. It should be noted that the term “block” or “video block” as used herein can be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to HEVC and VVC, for example, a block or video block can be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or can be or correspond to a respective block (e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0079] Prediction processing unit 41 can select one of a plurality of possible predictive encoding modes, e.g., one of a plurality of intra-predictive encoding modes or one of a plurality of inter-predictive encoding modes, for the current video block based on error results (e.g., rate and distortion levels). Prediction processing unit 41 can provide the resulting intra- or inter-predicted block to summer 50 to generate a residual block, and to summer 62 to reconstruct the encoded block for use as part of a reference frame at a later time. Prediction processing unit 41 also provides syntax elements, e.g., motion vectors, intra-mode indicators, partitioning information, and other such syntax information, to entropy encoding unit 56.

[0080] To select a suitable intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 can perform intra-prediction coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block being coded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 can perform multiple coding passes, e.g., to select a suitable coding mode for each block of video data.

[0081] In some implementations, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating motion vectors according to a predetermined pattern within a sequence of video frames, the motion vectors indicating displacement of video blocks within the current video frame relative to prediction blocks within a reference video frame. Motion estimation performed by motion estimation unit 42 is a process of generating motion vectors that estimate motion for video blocks. For example, a motion vector can indicate displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame that is related to a current block being coded within the current frame. The predetermined pattern can designate video frames in the sequence as P-frames or B-frames. Intra-BC unit 48 can determine vectors for intra-BC coding (e.g., block vectors) in a similar manner as motion vectors determined by motion estimation unit 42 for inter-prediction, or can utilize block vectors determined by motion estimation unit 42.

[0082] In terms of pixel difference, a prediction block for a video block can be or can correspond to a block or reference block of a reference frame that is deemed to closely match the video block being coded, the pixel difference can be determined by a sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metric. In some implementations, video encoder 20 can calculate values for sub-integer pixel positions of reference frames stored in DPB 64. For example, video encoder 20 can interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference frame. Thus, motion estimation unit 42 can perform a motion search relative to full-pixel positions and fractional-pixel positions and output motion vectors with fractional-pixel precision.

[0083] Motion estimation unit 42 calculates a motion vector for a video block in an inter-prediction coded frame by comparing a location of the video block to a location of a prediction block of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of the first and second reference frame lists identifying one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44, which then sends to entropy encoding unit 56.

[0084] Motion compensation performed by motion compensation unit 44 can involve fetching or generating a prediction block based on a motion vector determined by motion estimation unit 42. After receiving a motion vector for a current video block, motion compensation unit 44 can locate the prediction block pointed to by the motion vector in one of the reference frame lists, retrieve the prediction block from DPB 64, and forward the prediction block to summer 50. Summer 50 then forms a residual video block of pixel difference values by subtracting the pixel values of the prediction block provided by motion compensation unit 44 from the pixel values of the current video block being encoded. The pixel difference values forming the residual video block can include luma component differences or chroma component differences or both. Motion compensation unit 44 can also generate syntax elements associated with a video block of a video frame for use by video decoder 30 when decoding the video block of the video frame. The syntax elements can include, for example, syntax elements defining motion vectors used to identify a prediction block, any flags indicating a prediction mode, or any other syntax information described herein. It is noted that motion estimation unit 42 and motion compensation unit 44 can be highly integrated, but are illustrated separately for conceptual purposes.

[0085] In some implementations, intra BC unit 48 can generate vectors and fetch prediction blocks in a manner similar to that described above in connection with motion estimation unit 42 and motion compensation unit 44, but the prediction blocks are in the same frame as the current block being encoded, and the vectors are referred to as block vectors rather than motion vectors. Specifically, intra BC unit 48 can determine an intra prediction mode to use for encoding the current block. In some examples, intra BC unit 48 can encode the current block using various intra prediction modes, e.g., during a separate encoding pass, and test their performance through rate-distortion analysis. Next, intra BC unit 48 can select an appropriate intra prediction mode to use among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, intra BC unit 48 can compute rate-distortion values for the various tested intra prediction modes using rate-distortion analysis, and select the intra prediction mode with the best rate-distortion characteristics among the tested modes as the appropriate intra prediction mode to use. Rate-distortion analysis generally determines an amount of distortion (or error) between an encoded block and the original, unencoded block that was encoded to produce the encoded block, and a bit rate (i.e., number of bits) used to produce the encoded block. Intra BC unit 48 can compute a ratio from the distortion and rate for various encoded blocks to determine which intra prediction mode exhibits the best rate-distortion values for the block.

[0086] In other examples, intra BC unit 48 can perform such functions for intra BC prediction according to the implementations described herein using motion estimation unit 42 and motion compensation unit 44 in whole or in part. In either case, for intra block copy, in terms of pixel difference, the prediction block can be a block that is deemed to closely match the block to be encoded, the pixel difference can be determined by SAD, SSD, or other difference metric, and identifying the prediction block can include calculating values for sub-integer pixel positions.

[0087] Whether the prediction block is from the same frame according to intra prediction or from a different frame according to inter prediction, video encoder 20 can form pixel difference values by subtracting the pixel values of the prediction block from the pixel values of the current video block being encoded, thereby forming a residual video block. The pixel difference values forming the residual video block can include both luma component differences and chroma component differences.

[0088] As an alternative to inter prediction performed by motion estimation unit 42 and motion compensation unit 44 or intra block copy prediction performed by intra BC unit 48 as described above, intra prediction processing unit 46 can intra predict the current video block. In particular, intra prediction processing unit 46 can determine an intra prediction mode to use for encoding the current block. To this end, intra prediction processing unit 46 can encode the current block using various intra prediction modes, e.g., during a separate encoding pass, and intra prediction processing unit 46 (or, in some examples, a mode selection unit) can select an appropriate intra prediction mode to use from among the tested intra prediction modes. Intra prediction processing unit 46 can provide information indicative of the selected intra prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 can encode the information indicative of the selected intra prediction mode into the bitstream.

[0089] After prediction processing unit 41 determines a prediction block for the current video block via inter prediction or intra prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block can be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform, e.g., a discrete cosine transform (DCT) or a conceptually similar transform.

[0090] Transform processing unit 52 can send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce bit rate. The quantization process can also reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting a quantization parameter. In some examples, quantization unit 54 can then perform a scan on the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 can perform the scan.

[0091] Following quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), Probability Interval Partitioning Entropy (PIPE) coding, or another entropy encoding methodology or technique. The encoded bitstream can then be transmitted to video decoder 30 as shown in FIG. 3, or archived, such as in storage device 32 as shown in FIG. 3 for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 can also entropy encode motion vectors and other syntax elements for the current video frame that is being encoded. FIG. 1A FIG. 1A

[0092] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transforms, respectively, to reconstruct the residual video blocks in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 can generate motion compensated prediction blocks from one or more reference blocks of frames stored in DPB 64. Motion compensation unit 44 can also apply one or more interpolation filters to a prediction block to calculate sub-integer pixel values for use in motion estimation.

[0093] Adder 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reference block for storage in DPB 64. The reference block can then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a prediction block to inter predict another video block in a subsequent video frame.

[0094] FIG. 2B FIG. 3 is a block diagram illustrating another example video decoder 30 according to some embodiments of the present application. Video decoder 30 includes video data memory 79, entropy decoding unit 80, prediction processing unit 81, inverse quantization unit 86, inverse transform processing unit 88, adder 90, and DPB 92. Prediction processing unit 81 further includes motion compensation unit 82, intra prediction unit 84, and intra BC unit 85. Video decoder 30 can perform a decoding process generally reciprocal to the encoding process described above in connection with video encoder 20. FIG. 1G The decoding process described in connection with video encoder 20 is substantially reciprocal. For example, motion compensation unit 82 can generate prediction data based on motion vectors received from entropy decoding unit 80, while intra prediction unit 84 can generate prediction data based on intra prediction mode indicators received from entropy decoding unit 80.

[0095] ​​In some examples, the components of video decoder 30 can be tasked to perform the implementations of the present application. Moreover, in some examples, the implementations of the present disclosure can be distributed among one or more of the components of video decoder 30. For example, intra BC unit 85 can perform the implementations of the present application, alone or in combination with other units of video decoder 30, such as motion compensation unit 82, intra prediction unit 84, and entropy decoding unit 80. In some examples, video decoder 30 can not include intra BC unit 85, and the functionality of intra BC unit 85 can be performed by other components of prediction processing unit 81, such as motion compensation unit 82.

[0096] Video data memory 79 can store video data, such as an encoded video bitstream, to be decoded by the other components of video decoder 30. The video data stored in video data memory 79 can be obtained, for example, from storage device 32, from a local video source, such as a camera, via wired or wireless network communication of video data, or by accessing physical data storage media (e.g., a flash drive or hard disk). Video data memory 79 can include an encoded picture buffer (CPB) that stores encoded video data from an encoded video bitstream. DPB 92 of video decoder 30 stores reference video data for use in decoding video data by video decoder 30 (e.g., in intra- or inter-coding modes). Video data memory 79 and DPB 92 can be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magneto resistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For FIG. 2B illustrative purposes, video data memory 79 and DPB 92 are depicted as two distinct components of video decoder 30. But it will be readily apparent to one of ordinary skill in the art that video data memory 79 and DPB 92 can be provided by same memory device or separate memory devices. In some examples, video data memory 79 can be on-chip with other components of video decoder 30, or off-chip relative to those components.

[0097] During the decoding process, video decoder 30 receives an encoded video bitstream that represents encoded video frames and associated syntax elements. Video decoder 30 can receive the syntax elements at the video frame level and / or video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors, or intra-prediction mode indicators, among other syntax elements. Entropy decoding unit 80 then forwards the motion vectors, or intra-prediction mode indicators, among other syntax elements, to prediction processing unit 81.

[0098] When a video frame is coded as an intra-predicted (I) frame or an intra-coded prediction block in another type of frame, intra-prediction unit 84 of prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode signaled and reference data from previously decoded blocks of the current frame.

[0099] When a video frame is coded as an inter-predicted (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 produces one or more prediction blocks for a video block of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the prediction blocks can be produced from a reference frame within one of the reference frame lists. Video decoder 30 can construct the reference frame lists, i.e., List 0 and List 1, using default construction techniques based on reference frames stored in DPB 92.

[0100] In some examples, when a video block is coded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 produces a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block can be within a reconstructed region of the same picture as the current video block, as defined by video encoder 20.

[0101] Motion compensation unit 82 and / or intra BC unit 85 determine the prediction information for a video block of the current video frame by parsing the motion vectors and other syntax elements, and then use the prediction information to produce the prediction block for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode used to code the video block of the video frame (e.g., intra-prediction or inter-prediction), the inter-prediction frame type (e.g., B or P), the construction information for one or more of the reference frame lists for the frame, the motion vectors for each inter-predicted coded video block of the frame, the inter-prediction status for each inter-predicted coded video block of the frame, and other information used to decode the video block in the current video frame.

[0102] Similarly, intra BC unit 85 can use some of the received syntax elements, such as flags, to determine that the current video block is predicted using the intra BC mode, the construction information for which video blocks of the frame are within the reconstructed region and should be stored in DPB 92, the block vector for each intra BC predicted video block of the frame, the intra BC prediction status for each intra BC predicted video block of the frame, and other information used to decode the video block in the current video frame.

[0103] Motion compensation unit 82 can also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video blocks to calculate interpolated values for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 can determine the interpolation filters used by video encoder 20 from the syntax elements received and use these interpolation filters when generating the prediction blocks.

[0104] Inverse quantization unit 86 inverse quantizes quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80 using the same quantization parameter calculated by video encoder 20 for each video block in the video frame to determine a degree of quantization. Inverse transform processing unit 88 applies an inverse transform, e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to reconstruct the residual blocks in the pixel domain.

[0105] After motion compensation unit 82 or intra BC unit 85 generates the prediction block for the current video block based on the vectors and other syntax elements, adder 90 reconstructs the decoded video block for the current video block by adding the prediction block generated by motion compensation unit 82 and intra BC unit 85 to the residual block from inverse transform processing unit 88. In-loop filter 91, e.g., a de-blocking filter, a SAO filter, a CCSAO filter, and / or an ALF, can be located between adder 90 and DPB 92 to further process the decoded video block. In some examples, in-loop filter 91 can be omitted, and the decoded video block can be directly provided by adder 90 to DPB 92. The decoded video blocks in a given frame are then stored in DPB 92, which stores reference frames for subsequent motion compensation of video blocks that follow. DPB 92 or a memory device separate from DPB 92 can also store decoded video for later presentation on a display device (e.g., display device 34 of FIG. 1). FIG. 1A

[0106] In the current VVC and AVS3 standards, the motion information of the current coding block is either copied from a spatial or temporal neighboring block specified by a merge candidate index or obtained by explicit signaling of motion estimation. The focus of the present disclosure is to improve the accuracy of the motion vector of the affine merge mode by improving the derivation method of the affine merge candidate. To facilitate the description of the present disclosure, the existing affine merge mode design in the VVC standard is used as an example to illustrate the proposed ideas. Please note that although the existing affine mode design in the VVC standard is used as an example throughout the disclosure, the proposed techniques can also be applied to different designs of affine motion prediction modes or other coding tools with the same or similar design spirit for those skilled in the art of modern video coding technology.

[0107] ​In a typical video encoding and decoding process, a video sequence typically consists of an ordered set of frames or images. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of chroma samples (Cb). SCr is a two-dimensional array of chroma samples (Cr). In other instances, a frame may be monochromatic and therefore consist of only a two-dimensional array of luma samples.

[0108] like FIG. 1C As shown, the video encoder 20 (or more specifically, the partitioning unit in the predictive processing unit of the video encoder 20) generates an encoded representation of the frame by first partitioning the frame into a set of CTUs. A video frame may include an integer number of CTUs ordered consecutively from left to right and from top to bottom in raster scan order. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by the video encoder 20 in the sequence parameter set such that all CTUs in the video sequence have the same size, namely one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that this application is not limited to a specific size. FIG. 1D As shown, each CTU may include a luma sample CTB, two corresponding chroma sample coded tree blocks, and syntax elements for encoding and decoding the samples of the coded tree blocks. The syntax elements describe the properties of different types of units within the pixel coded block and how the video sequence can be reconstructed at the video decoder 30, including inter-frame prediction or intra-frame prediction, intra-frame prediction mode, motion vectors, and other parameters. In monochrome images or images with three separate color planes, the CTU may include a single coded tree block and syntax elements for encoding and decoding the samples of the coded tree block. The coded tree block can be an N×N sample block.

[0109] To achieve better performance, the video encoder 20 can recursively perform tree partitioning (such as binary tree partitioning, ternary tree partitioning, quadtree partitioning, or a combination thereof) on the coding tree blocks of the CTU, and divide the CTU into smaller CUs. FIG. 1E The diagram depicts the 64×64 CTU 400 first divided into four smaller CUs, each with a block size of 32×32. Within these four smaller CUs, CU 410 and CU 420 are each divided into four 16×16 CUs. These two 16×16 CUs, 430 and 440, are further divided into four 8×8 CUs. FIG. 1F The illustration depicts, as shown FIG. 1E The final result of the partitioning process of the CTU 400, as depicted in the diagram, is a quadtree data structure, where each leaf node of the quadtree corresponds to a CU with a corresponding size ranging from 32×32 to 8×8. Similar to... FIG. 1DEach CU can include one luma sample block and two corresponding chroma sample blocks (of the same size as the frame) and syntax elements used to code the sample of the coding block. In monochrome pictures or pictures with three separate color planes, a CU can include a single coding block and syntax structures used to code the sample of the coding block. It should be noted that, FIG. 1E to FIG. IF The quad-tree partitioning depicted in FIG. 1 is for illustration only and one CTU can be partitioned into multiple CUs to adapt to different local characteristics based on quad-tree / triple-tree / binary-tree partitioning. In multi-type tree structure, one CTU is partitioned by quad-tree structure and each quad-tree leaf CU can be further partitioned by binary-tree structure or triple-tree structure. As shown in FIG. 2, there are five possible partition types for a coding block with width W and height H, i.e., quad partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal triple partitioning, and vertical triple partitioning. FIG. 3A to FIG. 3E

[0110] In some embodiments, video encoder 20 can further partition the coding block of a CU into one or more (MxN) PBs. A PB is a rectangular (square or non-square) block of samples to which the same prediction (inter or intra) is applied. A PU of a CU can include a PB of luma samples, two corresponding PBs of chroma samples, and syntax elements used to predict the PBs. In monochrome pictures or pictures with three separate color planes, a PU can include a single PB and syntax structures used to predict the PB. Video encoder 20 can generate a predicted luma block, a predicted Cb block, and a predicted Cr block for the luma PB, the Cb PB, and the Cr PB of each PU of a CU.

[0111] Video encoder 20 can use intra prediction or inter prediction to generate the predicted block for a PU. If video encoder 20 uses intra prediction to generate the predicted block of a PU, video encoder 20 can generate the predicted block for the PU based on decoded samples of the frame associated with the PU. If video encoder 20 uses inter prediction to generate the predicted block of a PU, video encoder 20 can generate the predicted block for the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0112] ​After video encoder 20 generates the prediction luma blocks, the prediction Cb blocks, and the prediction Cr blocks for the PUs of a CU, video encoder 20 can generate a luma residual block for the CU by subtracting the prediction luma blocks of the CU from the original luma coded blocks of the CU, such that each sample in the luma residual block of the CU indicates a difference between a luma sample in one of the prediction luma blocks of the CU and a corresponding sample in the original luma coded blocks of the CU. Similarly, video encoder 20 can generate Cb residual blocks and Cr residual blocks for the CU, respectively, such that each sample in the Cb residual block of the CU indicates a difference between a Cb sample in one of the prediction Cb blocks of the CU and a corresponding sample in the original Cb coded blocks of the CU, and each sample in the Cr residual block of the CU can indicate a difference between a Cr sample in one of the prediction Cr blocks of the CU and a corresponding sample in the original Cr coded blocks of the CU.

[0113] Furthermore, as shown in FIG. 1E Video encoder 20 can use quad-tree partitioning to split the luma residual block, the Cb residual block, and the Cr residual block of a CU into one or more luma transform blocks, Cb transform blocks, and Cr transform blocks, respectively, as shown in FIG. 3. A transform block is a rectangular (square or non-square) block of samples that are applied the same transform. A TU of a CU can include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements for transforming the transform block samples. Thus, each TU of a CU can be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU can be a sub-block of the luma residual block of the CU. The Cb transform block can be a sub-block of the Cb residual block of the CU. The Cr transform block can be a sub-block of the Cr residual block of the CU. In monochrome pictures or pictures with three separate color planes, a TU can include a single transform block and syntax structures for transforming the samples of the transform block.

[0114] Video encoder 20 can apply one or more transforms to the luma transform block of a TU to generate a luma coefficient block for the TU. A coefficient block can be a two-dimensional array of transform coefficients. A transform coefficient can be a scalar. Video encoder 20 can apply one or more transforms to the Cb transform block of a TU to generate a Cb coefficient block for the TU. Video encoder 20 can apply one or more transforms to the Cr transform block of a TU to generate a Cr coefficient block for the TU.

[0115] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 can quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, providing further compression. After video encoder 20 quantizes a coefficient block, video encoder 20 can entropy encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 can perform CABAC on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 can output the bitstream that includes the bit sequence forming a representation of the encoded frame and associated data, which is saved in storage device 32 or transmitted to destination device 14.

[0116] After receiving the bitstream generated by video encoder 20, video decoder 30 can parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 can reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing the video data is generally reciprocal to the encoding process performed by video encoder 20. For example, video decoder 30 can perform an inverse transform on a coefficient block associated with a TU of a current CU to reconstruct a residual block associated with the TU of the current CU. Video decoder 30 also reconstructs the coding block of the current CU by adding the samples of the prediction block for the PU of the current CU to corresponding samples of the transform block of the TU of the current CU. After reconstructing the coding block for each CU of a frame, video decoder 30 can reconstruct the frame.

[0117] As described above, video coding primarily uses two modes (i.e., intra prediction (or intra-frame prediction) and inter prediction (or inter-frame prediction)) to achieve video compression. It should be noted that IBC can be considered as a third mode of intra prediction. Between the two modes, inter prediction contributes more to coding efficiency than intra prediction due to the use of motion vectors to predict a current video block from a reference video block.

[0118] However, as video data capture technology and finer video block sizes for preserving details in video data are constantly improving, the amount of data needed to represent motion vectors for a current frame also increases substantially. One way to overcome this challenge benefits from the fact that not only a group of neighboring CUs in both spatial and temporal domains have similar video data for prediction purposes, but also the motion vectors among these neighboring CUs are similar. Therefore, the motion information of spatial neighboring CUs and / or temporal co-located CUs can be used as an approximation of the motion information (e.g., motion vectors) of a current CU (which is also referred to as a “motion vector predictor” (MVP) of the current CU) by exploiting their spatial and temporal correlation.

[0119] Instead of using the motion information of spatial neighboring CUs and / or temporal co-located CUs as an approximation of the motion information (e.g., motion vectors) of a current CU (which is also referred to as a “motion vector predictor” (MVP) of the current CU) by exploiting their spatial and temporal correlation, as described above in connection with FIG. 1BThe actual motion vector of the current CU determined by the motion estimation unit is encoded into the video bitstream, a motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to generate a motion vector difference (MVD) for the current CU. By doing so, the motion vector determined by the motion estimation unit for each CU of the frame does not need to be encoded into the video bitstream, and the amount of data used to represent the motion information in the video bitstream can be significantly reduced.

[0120] Similar to the process of selecting a prediction block in a reference frame during inter prediction of a coding block, both video encoder 20 and video decoder 30 need to employ a set of rules for constructing a motion vector candidate list (also referred to as a "merge list") for a current CU using those potential candidate motion vectors associated with the spatial neighboring CUs and / or temporal collocated CUs of the current CU, and then selecting one member from the motion vector candidate list as the motion vector predictor for the current CU. By doing so, the motion vector candidate list itself does not need to be sent from video encoder 20 to video decoder 30, and the index of the selected motion vector predictor within the motion vector candidate list is sufficient for video encoder 20 and video decoder 30 to use the same motion vector predictor within the motion vector candidate list to encode and decode the current CU.

[0121] Some embodiments of the present disclosure aim to further improve inter coding efficiency by applying an adaptive refinement filter on the motion-compensated prediction signal of bi-predicted blocks. Some embodiments of the present disclosure aim to further improve the chroma coding efficiency of the motion compensation module applied in ECM. Some related coding tools applied in the transform and entropy coding process in ECM are briefly reviewed below. Subsequently, some deficiencies in the existing motion compensation design are discussed. Finally, a solution to improve the existing design is proposed.

[0122] Motion Compensated Prediction (MCP)

[0123] Motion-compensated prediction (MCP) (also known as motion compensation) is one of the most widely used video coding and decoding techniques in modern video coding and decoding standards. In MCP, a video frame is divided into multiple blocks (called prediction units (PUs)). Each PU is predicted from a block of the same size in a time reference picture, which significantly reduces the overhead required to transmit the block via signaling. In all existing video coding and decoding standards, each inter-frame PU is associated with a set of motion parameters, consisting of one or two MVs and a reference picture index. Inter-frame PUs in P-strips have only one list of reference pictures, while PUs in B-strips can use up to two lists of reference pictures. In MCP, the corresponding inter-frame prediction samples are generated based on the corresponding regions in the reference pictures identified by the MV and the reference picture index. The MV specifies the horizontal and vertical displacement between the current block and its reference block in the reference picture. FIG. 4 It shows where d x and d y This is an example of the horizontal and vertical values ​​of an MV. In reality, an MV value may have decimal precision. When an MV has a decimal value, an interpolation filter is applied to generate the corresponding predicted sample at the decimal sample location, such as... FIG. 5 As shown. VVC supports MV, which is 1 / 16 of the distance between two adjacent luminance samples in luminance MC, and MV, which is 1 / 32 of the distance between two adjacent chrominance samples in chrominance MC.

[0124] Adaptive Loop Filtering

[0125] In VVC and ECM, Adaptive Loop Filtering (ALF) selects one of 25 filters for each 4×4 block based on the direction and activity of the local gradient.

[0126] Filter shapes: Two diamond filter shapes were used (e.g. FIG. 6A to FIG. 6B (As shown). A 7×7 rhombus is used for the luminance component, and a 5×5 rhombus is used for the chrominance component.

[0127] Block Classification: For the luminance component, each 4×4 block is classified into one of 25 classes. The class index C is based on its directionality D and the quantization value of the activity. The results are as follows:

[0128]

[0129] To calculate D and First, the gradients in the horizontal, vertical, and two diagonal directions are calculated using the 1-D Laplacian operator:

[0130]

[0131] where the indices i and j refer to the coordinates of the top-left sample within the 4x4 block and R(i,j) denotes the reconstructed sample at coordinate (i,j). In order to reduce the complexity of the block classification, as shown in FIG. 7 a sub-sampled 1-D Laplacian computation is applied to the gradient computation in all directions.

[0132] The maximum and minimum of the gradients in the horizontal and vertical direction are then set to:

[0133]

[0134] The maximum and minimum of the gradients in the two diagonal directions are set to:

[0135]

[0136] In order to derive the value of the directionality D, these values are compared to each other and to two thresholds t1 and t2:

[0137] Step 1. If are both true, then D is set to 0.

[0138] Step 2. If then continue with step 3; otherwise, continue with step 4.

[0139] Step 3. If then set to 2; otherwise, D is set to 1.

[0140] Step 4. If then set to 4; otherwise, D is set to 3.

[0141] The activity value A is computed as:

[0142]

[0143] A is further quantized to the range of 0 to 4, inclusive, and the quantized value is denoted as For the chroma components in a picture, no classification method is applied.

[0144] Geometric Transforms of Filter Coefficients and Clip Values

[0145] Before filtering each 4x4 luma block, depending on the gradient values computed for the block, a geometric transform such as a rotation or a diagonal and vertical flip is applied to the filter coefficients f(k,l) and to the corresponding filter cutoff values c(k,l). This is equivalent to applying these transforms to the samples in the filter support region. The idea is to make the different blocks to which ALF is applied more similar by aligning their directionality.

[0146] Three geometric transforms are provided, including diagonal, vertical flip and rotation:

[0147]

[0148] where K is the size of the filter and 0≤k,l≤K-1 are the coefficient coordinates, with position (0,0) at the top-left corner and position (K-1,K-1) at the bottom-right corner. These transformations are applied to the filter coefficients f(k,l) and clipping values c(k,l), depending on the gradient values computed for the block. The relationships between the transformations and the four gradients for the four directions are summarized in Table 1 below.

[0149] Gradient Values Transform gd2< gd1 and gh < gv No Transform gd2< gd1 and gv< gh Diagonal gd1 < gd2 and gh < gv Vertical Flip gd1 < gd2 and gv < gh Rotation

[0150] Table 1

[0151] Filtering Process

[0152] When ALF is enabled for a CTB, each sample R(i,j) within the CU is filtered to obtain a sample value R'(i,j) as follows,

[0153]

[0154] where f(k,l) denotes the decoded filter coefficients, K(x,y) is a clipping function, and c(k,l) denotes the decoded clipping parameters. The variables k and l are located between and , where L denotes the filter length. Clip3(-y,y,x) is a clipping function that clips the input value x to the range [-y,y]. The clipping operation introduces a non-linearity that makes ALF more efficient by reducing the influence of neighboring sample values that differ too much from the current sample value.

[0155] Local Illumination Compensation

[0156] Local illumination compensation (LIC) is a coding tool studied in the VVC development process, aiming to address local illumination changes present in temporally neighboring pictures. LIC is based on a linear model in which a scaling factor and an offset are derived to enhance the prediction samples of the current block. Specifically, LIC can be mathematically modeled by the following equation:

[0157] P(x,y) = a - P r (x + v x ,y + v y ) + β (8)

[0158] where P(x,y) is the prediction signal of the current block at coordinates (x,y); P r (x + v x ,y + v y) is generated based on the motion vector (v x y ) of the current block and the motion vector (v FIG. 8 LIC process is illustrated. As shown, when LIC is applied to a video block, a linear model (i.e., scaling factor a and offset b) is derived by minimizing the difference between the neighboring samples (i.e., the template in FIG. 8 FIG. 8 FIG. 8

[0159] Since the scaling factor and offset are derived based on the current block and the template and its corresponding prediction signal, there is no need to signal the overhead of LIC parameters. In addition, for an inter frame block without merge, a LIC flag is signaled to indicate whether the LIC mode is enabled for the block. For a merge inter frame block, the LIC flag is considered as part of the motion information. Specifically, when constructing the merge list, in addition to the MV and reference index, the LIC flag will inherit the flag of its corresponding neighboring block. Meanwhile, LIC mode is also applied to affine inter frame blocks. When affine mode is applied, an inter frame block is divided into multiple sub-blocks and a specific MV is derived for each sub-block based on the affine model. Based on such design, when LIC is applied to an affine block, the corresponding LIC parameters will be derived based on the motion information of the sub-blocks on the top and left boundaries of the block; then, the derived LIC model is applied to the prediction samples of the whole block, as shown in FIG. 9. Since the MVs of the sub-blocks on the boundaries can be different, the prediction signal of the template is also generated on a sub-block basis and the prediction sample of each template sub-block is generated using the MV of the corresponding sub-block on the boundary of the coded block.

[0160] Finally, it is mentioned that in the current LIC design, LIC is only applied to inter frame blocks with single directional prediction.

[0161] Bi-prediction with CU-level weights

[0162] In HEVC, the bi-prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bi-prediction mode goes beyond simple averaging, allowing a weighted average of the two prediction signals, i.e.,

[0163] P bi-pred = ((8 - w) * P0 + w * P1 + 4) » 3 (9)

[0164] ​​​​Weighted bi-prediction allows five weights, w e {-2, 3, 4, 5, 10}. For each bi-predicted CU, the weight w is determined by one of the following two ways: 1) for non-merge CU, the weight index is signaled; 2) for merge CU, the weight index is inherited from one of the neighboring blocks based on the merge candidate index. In addition, in VVC, all 5 weights are used for low-delay pictures (i.e., all reference pictures have display order before the current picture). While for non-low-delay pictures (at least one reference has display order after the current picture), only 3 weights (w e {3, 4, 5}) are used.

[0165] Overlapped Block Motion Compensation

[0166] OBMC is a coding technique to remove block artifacts in the MC stage. The basic idea of OBMC is to perform motion compensation on the current block using MVs from neighboring blocks and combine multiple prediction signals using the neighboring MVs to generate the final prediction signal for the CU. For each inter CU, OBMC is performed on the top and left boundaries of the block. In addition, when a video block is coded in a sub-block mode (e.g., affine, ATMVP or DMVR), OBMC is also performed on all internal boundaries (i.e., top, left, bottom and right boundaries) of each sub-block. FIG. 15 The OBMC process applied to a CU without sub-block level motion compensation is illustrated. When OBMC is applied to a sub-block (e.g., sub-block A in FIG. 15 , the MVs of the neighboring sub-blocks to the right and bottom of the current sub-block are used to derive the prediction signal in addition to the left and top neighboring sub-blocks of the sub-block; then the four prediction blocks are averaged to generate the final prediction signal for the current sub-block.

[0167] In the current ECM software, a template-based OBMC scheme is applied. Specifically, the prediction value derivation method for the CU boundary samples is not using fixed weights to combine multiple motion compensation hypotheses, but is determined according to template matching cost, including using only the motion information of the current block, or also using the motion information of the neighboring blocks and adopting one of the hybrid modes.

[0168] In this scheme, for each 4x4 block on the top CU boundary, the template size is equal to 4x1. If N neighboring blocks have the same motion information, the template size is enlarged to 4Nx1 because the MC operation can be processed simultaneously. For each 4x4 left block at the left CU boundary, the left template size is equal to 1x4 or 1x4N (as shown in FIG. 16 ).

[0169] For each 4x4 top block (or N 4x4 block groups), the prediction value of the boundary samples is derived according to the following steps.

[0170] Take the current block A and its above neighboring block AboveNeighbor_A as an example. The left block operates in the same way.

[0171] First, three template matching costs (Cost1, Cost2, Cost3) are measured by SAD between the template reconstructed samples from MC process and their corresponding reference samples according to the following three categories of motion information:

[0172] Cost1 is calculated according to the motion information of A.

[0173] Cost2 is calculated according to the motion information of AboveNeighbor_A.

[0174] Cost3 is calculated according to the weighted prediction of the motion information of A and AboveNeighbor_A, with the weighting factors being 3 / 4 and 1 / 4 respectively.

[0175] Second, a method is selected to calculate the final prediction result of the boundary samples by comparing Cost1, Cost2 and Cost3.

[0176] The original MC result using the motion information of the current block is denoted as Pixel1, and the MC result using the neighboring block motion information is denoted as Pixel2. The final prediction result is denoted as NewPixel.

[0177] If Cost1 is the smallest, then NewPixel(i,j) = Pixel1(i,j).

[0178] If (Cost2 + (Cost2 » 2) + (Cost2 » 3)) <= Cost1, then use hybrid mode 1.

[0179] For luma blocks, the number of mixed pixel rows is 4.

[0180] NewPixel(i,0) = (26*Pixel1(i,0) + 6*Pixel2(i,0) + 16) » 5

[0181] NewPixel(i,1) = (7*Pixel1(i,1) + Pixel2(i,1) + 4) » 3

[0182] NewPixel(i,2) = (15*Pixel1(i,2) + Pixel2(i,2) + 8) » 4

[0183] NewPixel(i, 1) = (31 x Pixel1(i, 1) + Pixel2(i, 1) + 16) » 5

[0184] For chroma blocks, the number of mixed pixel rows is 1.

[0185] NewPixel(i, 0) = (26 x Pixel1(i, 0) + 6 x Pixel2(i, 0) + 16) » 5

[0186] If Cost1 <= Cost2, then use blend mode 2.

[0187] For luma blocks, the number of mixed pixel rows is 2.

[0188] NewPixel(i, 0) = (15 x Pixel1(i, 0) + Pixel2(i, 0) + 8) » 4

[0189] NewPixel(i, 1) = (31 x Pixel1(i, 1) + Pixel2(i, 1) + 16) » 5

[0190] For chroma blocks, the number of mixed pixel rows / columns is 1.

[0191] NewPixel(i, 0) = (15 x Pixel1(i, 0) + Pixel2(i, 0) + 8) » 4

[0192] Otherwise, use blend mode 3.

[0193] For luma blocks, the number of mixed pixel rows is 4.

[0194] NewPixel(i, 1) = (7 x Pixel1(i, 1) + Pixel2(i, 1) + 4) » 3

[0195] NewPixel(i, 2) = (15 x Pixel1(i, 2) + Pixel2(i, 2) + 8) » 4

[0196] NewPixel(i, 3) = (31 x Pixel1(i, 3) + Pixel2(i, 3) + 16) » 5

[0197] For chroma blocks, the number of mixed pixel rows is 1.

[0198] NewPixel(i, 0) = (7 x Pixel1(i, 0) + Pixel2(i, 0) + 4) » 3

[0199] Merge candidate adaptive reordering with template matching

[0200] In ECM, a reordering tool called adaptive reordering of merge candidates with template matching (ARMC) is applied to inter-coded merge mode. When this method is applied, the merge candidates are adaptively ordered according to the template matching (TM) cost. This method is applicable to both regular merge mode and affine merge mode.

[0201] Specifically, in the ARMC design, an initial merge candidate list is first constructed, which includes multiple merge candidates, such as spatial, TMVP, non-adjacent, HMVP, and pair-wise merge candidates. The candidates in the initial list are then divided into one or more subgroups. The merge candidates in each subgroup are reordered according to the cost based on template matching to generate a reordered merge candidate list. Then, the indices of the selected merge candidates in the reordered merge candidate list are signaled from the encoder to the decoder.

[0202] During the reordering process, the template matching cost of a merge candidate is measured by the SAD between the samples of the template of the current block and their corresponding reference samples. The template includes a set of reconstructed samples neighboring the current block. The reference samples of the template are located by the motion information of the merge candidate. When the merge candidate utilizes bi-prediction, the reference samples of the template of the merge candidate are also generated by bi-prediction, as shown in FIG. 18 .

[0203] For affine mode, since different sub-blocks can represent different motion vectors, the predicted samples of the template are generated according to sub-block based motion compensation. Specifically, as shown in FIG. 19 , assuming the sub-block size of the sub-block based affine merge candidate is equal to Wsubx Hsub, the top template includes several sub-templates with size Wsubx 1, and the left template includes several sub-templates with size 1x Hsub. The motion information of the sub-blocks in the first row and the first column of the current block is used to derive the reference samples of each sub-template.

[0204] Merge mode with MVD based on template matching

[0205] In addition to the merge mode in which the implicitly derived motion information is directly used for the prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is applied to both regular merge mode and affine merge mode. For signaling, an MMVD flag is signaled immediately after the regular merge flag to specify whether the MMVD mode is used for the CU. In ECM, 16 refinement locations along kx p / 8 diagonal angles are defined for the MMVD mode, as shown inFIG. 20 The top N motion candidates in the candidate list before reordering are used as the base candidates for MMVD and affine MMVD. For MMVD, N is equal to 3; while for affine MMVD, N is equal to 1 or 3 depending on the affine flag of the neighboring block. When one base candidate is bi-predicted, two ways of adding MMVD offsets are allowed, including 'double-side' and'single-side'. In 'double-side' MMVD mode, the same selected MMVD offset (or its opposite value) is applied to the L0 and L1 MVs of the candidate depending on the POC relationship of the current picture with its reference pictures in L0 and L1. In'single-side' MMVD mode, the selected MMVD offset is only applied to the MV in one reference picture list (L0 and L1), while the MV in the other reference list remains unchanged. Accordingly, based on such design, MMVD mode shares 16 x 6 x 3 = 288 refinement positions. To save the signaling overhead, the SAD cost between the template (one row above and one column left of the current block) of each refinement position and its reference is used to reorder all the possible 288 refinement positions, and only the top 36 refinement positions after reordering are allowed to be selected to be indicated by one MMVD index in the bitstream.

[0206] In all existing video coding standards, MCP plays a key role in ensuring the inter coding efficiency. With MCP, the video signal to be coded can be predicted from the temporal neighboring signals, and only the prediction error, MVs and reference picture indices are transmitted. As analyzed before, ALF can effectively improve the quality of the reconstructed video, and thus improve the performance of inter coding by providing high quality reference pictures. LIC can be considered as an enhancement of regular motion compensated prediction. Although both tools can improve the inter coding efficiency, the quality of temporal prediction can still not be good enough due to the following reasons

[0207] Video signals can be coded with coarse quantization (i.e., high quantization parameter (QP) values). When coarse quantization is applied, the reconstructed pictures can contain severe coding artifacts, such as block artifacts, ringing artifacts, etc. Given that the reconstructed signal of the current picture will be used as the reference for temporal prediction, such distortion can degrade the effectiveness of MCP, and thus the inter coding efficiency of the subsequent pictures.

[0208] Although LIC can efficiently compensate the illumination changes between different pictures, it can only be applied to uni-predicted blocks. It is well known that the combination of multiple predicted blocks can efficiently suppress the coding noise (which is caused by the quantization / dequantization process) present in the motion compensated signal. Therefore, bi-predicted blocks are generally more compressively efficient than uni-predicted blocks, i.e., there are more bi-predicted blocks than uni-predicted blocks. This means that uni-prediction LIC cannot fully exploit the coding gain that can be achieved with the LIC tool.

[0209] According to the existing OBMC design in ECM, OBMC is always disabled for inter CUs coded using LIC. Since there are also blocking artifacts between inter blocks coded with LIC and inter blocks coded without LIC, this design is not optimal in terms of coding efficiency. Moreover, even in the case that both neighboring blocks apply LIC, there can be potential blocking artifacts at the block boundary of these two blocks since the LIC parameters applied to these two blocks can be different.

[0210] Methods and devices are proposed in this disclosure to improve the efficiency of motion compensation, thus improving the quality of temporal prediction. Specifically, it is proposed to apply adaptive filtering at the prediction samples of bi-predicted blocks. To reduce the signaling overhead, the filter coefficients are derived from the neighboring reconstructed samples (i.e., the template) of the current block and their corresponding prediction samples. In this way, the energy of the prediction residual is mitigated, thus reducing the overhead of residual signal transmission.

[0211] FIG. 10 Error! Reference source not found. A block diagram of a video encoder applying the proposed adaptive bi-prediction filtering is given. First, similar to a conventional video encoder, the motion estimation and compensation module generates a motion compensated signal by matching the current block with one block (uni-prediction) or two blocks (bi-prediction) in the reference picture using the best MVs. Then, for bi-predicted blocks, the motion compensated samples (luma and chroma) are provided to the proposed adaptive filter to generate the filtered motion compensated prediction samples of the current block. Then, the original signal is subtracted from the prediction signal to remove the temporal redundancy and produce the corresponding residual signal. The residual signal is transformed and quantized, then entropy coded and output to the bitstream. To obtain the reconstructed signal, the residual signal is reconstructed by inverse quantization and inverse transformation. Then, the reconstructed residual is added to the motion compensated prediction. Further, loop filtering processes (e.g., deblocking, ALF and SAO) are applied to the reconstructed video signal for output. As will be discussed later, the filter coefficients of the proposed adaptive bi-prediction filter are derived directly from the neighboring reconstructed luma and chroma samples at the decoder. In addition, to maximize the coding gain of the proposed method, additional syntax can be signaled at a given block level (e.g., CTU, CU or PU level) to indicate whether the proposed filtering is applied to the current block for motion compensation.

[0212] FIG. 11 A block diagram of the proposed decoder is shown, which receives the bitstream generated by the encoder FIG. 10The encoder in the encoder produces a bitstream. At the decoder, the bitstream is first parsed by the entropy decoder. Then the residual coefficients are dequantized and inverse transformed to obtain the reconstructed residual. For temporal prediction, the prediction signal is first generated by obtaining the motion compensated block using the signaled prediction information (i.e., MV and reference index). Then, for bi-predicted blocks, it is parsed from the bitstream to determine whether the adaptive filter is enabled for this block. If the adaptive filter is enabled, the motion compensated luma and chroma signals will be further processed by the proposed adaptive filter; otherwise, the motion compensated chroma signals will not be filtered. Then, the motion compensated signal (filtered or unfiltered) and the reconstructed residual are added to obtain the reconstructed video. The reconstructed video can also need to go through in-loop filtering before being stored in the reference picture store for display and / or used to decode future video signals.

[0213] Adaptive bi-prediction filtering based on template bi-predicted samples

[0214] In this section, an adaptive filter scheme for bi-predicted blocks is proposed, in which the filter coefficients are derived based on the bi-predicted samples of a template of the bi-predicted block. Specifically, in the proposed scheme, the bi-predicted samples of the template are first generated according to the motion vectors of the current block; then, a least mean square error (LMSE) algorithm is applied to derive the filter parameters by minimizing the difference between the template predicted samples and the template samples. FIG. 12 The proposed adaptive filter method based on the bi-predicted samples of a template is illustrated. As shown in FIG. 1, T represents the template of the current bi-predicted block; T0and T1are the L0and L1predicted samples of the template, which are generated using the bi-predicted motion vectors of the current block, i.e., T0= L0(T) and T1= L1(T). Based on these notations, in the proposed scheme, the bi-predicted samples of the template are first generated by averaging the two uni-predicted samples of the template in L0and L1, i.e., FIG. 12 and

[0215] T bi = w0*T0+ w1*T1 (10)

[0216] where w0and w1are the weights applied to the L0and L1directions when generating the bi-predicted samples of the current block, which are equal to 0.5 when no BCW is applied, and can be equal to -0.125, 0.375, 0.625 and 1.125 when BCW is applied. Based on the derived bi-predicted samples of the template, the coefficient values of the adaptive filter are calculated by minimizing the difference between the template samples and their bi-predicted samples using LMSE derivation, i.e.,

[0217]

[0218] where f​​* denotes the filter applied to a template prediction sample T bi (x,y) of a corresponding HxL neighboring region, where, In practice, various filters with different sizes and shapes can be applied to provide different trade-offs between coding performance and complexity. Larger filters can make the template prediction samples closer to the template samples, but at the cost of increased computational complexity. Finally, the derived filter coefficients are applied to modify the original bi-prediction signal of the current block as follows

[0219]

[0220] where P bi (x,y) and P′ bi (x,y) are the bi-prediction samples before and after applying the proposed adaptive filter. In addition, to further improve the coding gain, the proposed method introduces an offset and some non-linear terms when deriving the filter coefficients, which can further reduce the distortion between the template samples and their prediction samples. Specifically, with such modification, the filter coefficient derivation in (11) becomes

[0221]

[0222] and the filter application in (12) becomes

[0223]

[0224] where o is the offset, nl k is the non-linear term, which is represented as a summation of a series of powers of a template prediction sample T bi (2x,2y) (i.e., k = 2, …, K-1).

[0225] In one or more examples, a linear model (i.e., scaling factor and offset) is proposed to derive a bi-tap filter to enhance the prediction samples of a bi-predicted block. Specifically, a bi-predictive LIC is proposed, which operates as follows: 1) generate the bi-prediction samples of the template as shown in (10); 2) derive the scaling factor and offset using the template samples and their corresponding bi-prediction samples as shown below

[0226]

[0227] where a and b are the scaling factor and offset of the LIC linear model; N is the number of template samples involved in the derivation. After that, the final bi-prediction of the current block is generated as shown below

[0228] P′ bi (x,y) = a · Pbi (x, y) + β (16)

[0229] Adaptive bi-prediction filtering based on template uni-predicted samples

[0230] In this section, an adaptive bi-prediction filtering scheme for uni-prediction samples using a template of a bi-predicted block is proposed. For example, in this method, two adaptive filtering operations are applied to the prediction samples of the template in a uni-directional manner: two sets of filter coefficients are derived and applied to the prediction samples in L0 and L1, respectively; then a weighted average of the two filtered uni-predicted samples is formed as the final prediction sample of the current block. FIG. 13 The proposed scheme is illustrated. As FIG. 13 shown, two uni-predictions T0 and T1 of the template are generated based on the L0 and L1 MVs. Then, based on separate minimization of the distortion between T0 and T and T1 and T, two sets of filter parameters f0 and f1 can be derived for the L0 and L1 directions, respectively, as follows:

[0231]

[0232] where N denotes the number of template samples involved; T is the template samples of the current block; denotes the uni-prediction of the template samples based on the MV of the current block (L0 or L1). After that, the two filters are applied to the two uni-predictions of the current block, respectively, and then they are combined to generate the final bi-prediction of the current block, as follows

[0233] P' bi (x, y) = w0*P'0(x, y) + w1*P'1(x, y) (18)

[0234] where

[0235]

[0236] where P0(x, y) and P1(x, y) are the two uni-predicted samples of the current block before the proposed adaptive filtering is applied. Similar to (13) and (14), in addition, to further improve the coding gain, an offset and a non-linear term can be introduced when deriving the filter coefficients. With such modifications, the filter coefficients can be derived as follows

[0237]

[0238] and the filtered uni-predicted samples of the current block are calculated as

[0239]

[0240] In one or more examples, it is proposed to use linear models (i.e., scaling factors and offsets) to derive a bi-prediction filter to enhance two uni-predictions of a bi-predicted block. Specifically, a bi-predictive LIC is proposed, which operates as follows: 1) generate two uni-predictions of the template; 2) derive two sets of scaling factors and offsets using the template samples and their corresponding uni-predicted samples, as follows

[0241]

[0242] where a0and b0are the scaling factors and offsets of the LIC linear model of the L0 uni-prediction, and a1and b1are the scaling factors and offsets of the LIC linear model of the L1 uni-prediction; N is the number of template samples involved in the derivation. After that, the final bi-prediction of the current block is generated, as follows

[0243] P' = w0*P0+ w1*P1(22) bi (x, y) = w0*(a0-P0(x, y) + b0) + w1*(a1-P0(x, y) + b1) (23)

[0244] where w0and w1are the BCW weights applied to the current block.

[0245] Recursive uni-prediction filtering based on adaptive bi-prediction filtering

[0246] In FIG. 13 , since the filter coefficients applied to the two uni-prediction signals of the template are derived separately, the resulting bi-prediction signal of the template (i.e., the weighted combination of the two filtered uni-prediction signals) can not be optimal when considering the minimization of the distortion between the template samples and their corresponding predicted samples. To solve this problem, an iterative scheme is proposed to derive the optimal filter coefficients applied to the two uni-prediction signals of the template of a bi-predicted block. The proposed scheme proceeds in an iterative manner, alternating the optimization of the prediction filter of one prediction direction while keeping the prediction filter of the other prediction direction fixed. Specifically, the derivation process of the two uni-prediction filter coefficients is summarized as follows:

[0247] Step 1: Given the starting prediction direction L (0) , derive the initial filter coefficients of the starting prediction direction by minimizing the distortion between the uni-prediction and the template T i.e.,

[0248]

[0249] Step 2: Based on the filter coefficients , compute the filtered uni-prediction and set k = 1.

[0250]

[0251] Step 3: Select the target prediction direction L (k) = 1 - L (k-1) and compute the target template samples of the current block as follows

[0252]

[0253] Step 4: Derive the filter coefficients of the starting prediction direction L by minimizing the distortion between the unidirectional prediction (k) and the template T (k) , i.e.,

[0254]

[0255] Step 5: Compute the filtered unidirectional prediction based on the filter coefficients as follows

[0256]

[0257] Step 6: Set k = k + 1 and go to Step 3.

[0258] The resulting filter is used as the corresponding filter applied to the two unidirectional predictions of the current block, and then the filtered prediction samples are combined to generate the final bi-directional prediction of the current block as shown in (18) and (19). Similarly, the offset and non-linear terms as shown in (20) and (21) can also be applied in the proposed iterative bi-directional prediction filter derivation scheme. In addition, in one or more examples, it is proposed to derive a bi-tap filter by the proposed iterative filter derivation scheme using a linear model (i.e., a scaling factor and an offset): 1) generate the two unidirectional predictions of the template; 2) derive two sets of scaling factors and offsets based on the iterative algorithm shown in Steps 1-6; 3) compute the final bi-directional prediction samples of the current block as shown in (23).

[0259] ​In practical applications, different number of iterations can be applied to the above-mentioned iterative filter derivation schemes. Generally speaking, the more iterations are applied, the smaller the distortion between the template and its predicted signal (i.e., the better coding gain), but at the cost of increased computational complexity. Different approaches are proposed to determine the number of iterations to be applied in the proposed algorithms. In one approach, it is proposed to use a fixed number of iterations (i.e., 3) at both the encoder and the decoder. In a second approach, it is proposed to let the encoder freely choose the specific number of iterations and signal the corresponding value to the decoder. When this approach is applied, a new syntax element(s) can be added in the sequence parameter set (SPS), picture parameter set (PPS), picture header, slice header, or even at the coding block level to indicate the value of the applied iterations. In a third approach, it is proposed to adaptively determine the number of iterations to be applied to a block based on the statistics of the block (e.g., sample variation, motion vector difference, and additional information). In one or more examples, it is proposed to use the difference between the original L0 and L1 prediction samples of a bi-predicted block as the criterion to choose the number of iterations to be applied. For example, when the difference (i.e., sum of absolute difference (SAD), sum of squared difference (SSD), and other matrix) between the two prediction samples is larger than a threshold, a larger number of iterations is applied to the block; otherwise (i.e., the difference is smaller than the threshold), a smaller number of iterations is applied.

[0260] Last but not least, different initial prediction directions can be applied in the proposed schemes. In one approach, it is proposed to always use L0 as the initial prediction direction in the proposed methods. In another approach, it is proposed to use L1 as the initial prediction direction. In a third approach, it is proposed to select the initial prediction direction based on the slice type, prediction structure, and QP of the slice to which the current block belongs. For example, L0 can be used as the initial prediction direction for non-low-delay pictures, while L1 can be used as the initial prediction direction for low-delay pictures.

[0261] Adaptive motion compensation filtering based on non-adjacent spatial neighboring blocks

[0262] In some embodiments, the blocks surrounding the current block are defined as the neighboring blocks of the current block. As FIG. 17A and FIG. 17BAs shown, the blank neighboring blocks without shades are considered adjacent neighboring blocks, while those with shades are considered non-adjacent neighboring blocks. In the above method, the coefficients of the proposed motion compensation filter are always derived from the reconstructed samples adjacent to the current coding block (i.e., the neighboring samples immediately above and to the left). This scheme can be efficient when the current block is highly correlated with its adjacent spatial neighboring blocks. However, in real-world encoding / decoding scenarios, due to encoding / decoding noise (e.g., noise caused by quantization / dequantization and block artifacts introduced during the motion compensation stage), the current block may be more correlated with samples in the reconstructed regions that are not adjacent to the current block. Based on this consideration, this section proposes an adaptive motion compensation filtering scheme based on non-adjacent neighboring blocks. Using this scheme, the coefficients of the adaptive motion compensation filter can be derived using samples in non-adjacent regions. Different methods can be applied to locate the non-adjacent reconstructed samples used to derive the filter coefficients. In one or more embodiments, non-adjacent neighboring blocks can be scanned from the left and top regions of the current block. The scan distance can be defined as the number of scan block sizes from the left or top of the current block.

[0263] As shown in Figure 17, multiple rows (columns) of non-adjacent neighboring blocks can be scanned at the top or left of the current block. The distances shown in Figure 17 represent the number of scan block sizes from each candidate location to the current block, with each scan block size representing one distance unit. For example, a region to the left of the current block with a "distance 2" indicates that candidate neighboring blocks located in that region are 2 scan block sizes away from the current block. Based on this pattern, different scan block sizes can be applied:

[0264] In one method, such as FIG. 17A As shown, non-adjacent neighboring blocks at each distance can have the same block size as the current block. Note that when this method is applied, the granularity of the block scan is adaptively adjusted according to the partition granularity of the current block; that is, larger coded blocks have a greater chance of utilizing more distant non-adjacent reconstructed samples to compute the coefficients of the adaptive filter.

[0265] In another approach, non-neighboring blocks that can be used for filter coefficient derivation can be defined based on fixed blocks (e.g., 4×4, 8×8).

[0266] In the third method, a combination approach can be used to define the scan pattern. For example, for small blocks, a fixed scan block size (Ws × Hs) can be applied, where Ws and Hs are the width and height of the fixed scan block size; while for large blocks, the scan block size is defined as the current block size. Specifically, let xStep and yStep represent the width and height of the scan block size, with corresponding values ​​of xStep = max(Ws, width) and yStep = max(Hs, height), where the width and height are the width and height of the current block, respectively.

[0267] To indicate the use of non-adjacent neighboring blocks for filter derivation, the spatial candidate list can be formed to include both the adjacent neighboring blocks (i.e., the spatial neighboring reconstructed samples next to the top and left side) and the non-adjacent neighboring blocks. In some embodiments, an index can be signaled from the encoder to the decoder to specify which spatial candidate to select to derive the filter coefficients.

[0268] Additionally or alternatively, in some examples, it is proposed to apply the proposed non-adjacent spatial neighboring blocks to the existing LIC design, where the proposed adaptive motion compensated filter degrades to a 2-tap filter (i.e., one scaling and one offset). Specifically, based on the motion information of the current block (uni-prediction or bi-prediction), the method uses the motion information to generate the corresponding prediction signals of the selected non-adjacent blocks, which are then used to derive the corresponding LIC parameters by minimizing the difference between the reconstructed samples of the non-adjacent blocks and their corresponding predictions.

[0269] Adaptive motion compensation filtering based on history filter coefficients

[0270] In the above non-adjacent neighboring block based schemes, the filter coefficients are derived from the reconstructed area far away from the current block, which requires additional on-chip memory to store these non-adjacent reconstructed samples. This is relatively costly for practical hardware codec implementation. Therefore, in order to reduce the implementation cost, a history based adaptive motion compensation filtering method is proposed. In this method, the filter coefficients of a previously coded block are stored in a table and can be used for the filtering of the motion compensated samples of future blocks. In some embodiments, the table can be a candidate filter list. The table with multiple sets of filter coefficients can be maintained and synchronized in the encoding process and the decoding process. Whenever an inter block is coded, a set of filter coefficients can be derived based on its reconstructed samples and its predicted samples, and then added as a new candidate to the last entry of the table. In order to maintain the size of the table, a first-in-first-out (FIFO) rule can be used, in which a redundancy check can be applied to check whether there is a same candidate in the table as the new candidate. If yes, the same candidate will be removed from the table and all other candidates will be moved forward and the new candidate is added at the last entry. When the table is full and there is no same candidate in the table, the first candidate will be removed from the table and the new candidate is added at the last position. Then, a candidate set of filter coefficients can be selected for the filtering of the motion compensated samples of future coded blocks. For signal transmission, when the history based filter coefficient derivation is selected, an index can be signaled to indicate which candidate set in the table will be used to derive the filter coefficients of the current block. In another embodiment, in order to reduce the number of filter coefficient derivation, it is proposed to only include the filter coefficients of the coded blocks that have adaptive motion compensation filtering selected into the table.

[0271] Additionally or alternatively, in some examples, it is proposed to apply the proposed history based filter derivation scheme to the existing LIC design, in which the proposed adaptive motion compensation filtering degenerates to a 2-tap filter. Specifically, in this case, each candidate in the table is composed of two parameters, i.e., a scaling and an offset, which can be selected by an inter coded block to adjust its predicted samples.

[0272] Combination of adaptive motion compensation filtering and OBMC

[0273] In this section, methods of applying the proposed adaptive motion compensation filtering method to the OBMC process are provided. Specifically, in some example methods, it is proposed to consider the impact of the LIC parameters of each neighboring block on its corresponding motion compensated predicted samples when performing the OBMC process on the current block in addition to the motion vectors of the neighboring blocks. For ease of description, the proposed methods are illustrated below with the example of regular inter prediction without sub-block partitioning. For example, let P obmc(x,y) represents the mixed prediction sample at coordinates (x,y) after combining the prediction signal of the current CU with multiple prediction signals of the MV based on its spatial neighboring blocks. cur (x,y) represents the predicted sample point at the current CU coordinates (x,y); P top (x,y) and P left (x,y) represents the predicted sample points located at the same position as the current CU but using the MV of the left and right neighboring blocks of the CU, respectively. In some embodiments, as shown in equation (29), P obmc (x, y) can be P cur (x,y), P top (x,y) and P left The weighted average of (x,y).

[0274] P obmc (x,y)=w cur *P cur (x,y)+w top *P top (x,y)+w left *P left (x,y) (29)

[0275] Furthermore, for ease of explanation, assume that adaptive motion compensation filtering is applied to the current block and its top and left spatially neighboring blocks, and that the applied filter is a tapped filter (i.e., a scaling factor and an offset), with filter coefficients as follows: For the current block, α... cur and β cur The top neighboring block is α top and β top And the left neighbor is α left and β left The proposed scheme first generates predicted samples for the current block, as shown below.

[0276] P cur (x,y)=α cur ·P org cur (x,y)+β cur (30)

[0277] Among them, P org cur (x,y) are the raw prediction samples of the current block using the current block's motion vector without applying filtering. Then, the boundary prediction samples of the current CU are updated using the MVs of the top and left causal neighboring blocks of the current CU. First, the top neighboring block of the current block is checked. If the block is an inter-frame block, its MV and filter coefficients (i.e., α) are... top and β top) will be assigned to the current block to generate the prediction signal P at the collocated position of the current block top (x, y), as shown below.

[0278] P top (x, y) = a top · P org top (x, y) + b top (31)

[0279] where P org top (x, y) is the original prediction sample of the current block using the motion vector of the top neighboring block and without applying the filter. Then, following the same procedure, the corresponding prediction sample is generated based on the motion vector of the left neighboring block and the LIC parameters, as shown below

[0280] P left (x, y) = a left · P org left (x, y) + b left (32)

[0281] where P org left (x, y) is the original prediction sample of the current block using the motion vector of the left neighboring block and without applying the filter. Finally, these three prediction signals are combined according to the template-based OBMC blending process (as shown in the section of “Overlapped Block Motion Compensation”) to generate the final prediction sample of the current block.

[0282] When the current block is coded in a sub-block mode (e.g., affine, ATMVP and DMVR), the proposed OBMC based on motion compensation filtering can also be applied to the internal OBMC of the sub-blocks inside the current CU. Specifically, when applying this scheme, the same filtering process as shown in equations (29) to (31) can be applied to generate the corresponding prediction sample of each sub-block using the top, left, bottom and right neighboring sub-blocks of the sub-block. However, in the prediction sample derivation of the internal OBMC process, the filter coefficients of the current CU will always be applied instead of the LIC parameters of the spatial neighboring blocks.

[0283] To achieve different complexity / performance trade-offs, two methods of applying the proposed motion-compensated filtering OBMC are presented in this disclosure. In one method, it is proposed to only apply the filter-based OBMC to the prediction samples on the CU boundary, but not to the prediction samples of the sub-blocks inside the CU (i.e., intra OBMC). In this case, for intra OBMC, only the neighboring motion vectors of the neighboring blocks of each sub-block are considered to generate its OBMC prediction samples. In another method, it is proposed to apply the filter-based OBMC to the prediction samples on the CU boundary as well as along the sub-block boundaries of the sub-blocks inside the CU.

[0284] Combination of adaptive motion compensation filtering and inter tools based on template matching

[0285] As discussed in the “Introduction” section, several template matching based techniques are introduced in the ECM to reduce the merge mode’s preservation overhead. For example, in ARMC, the candidates in the initial merge candidate list are divided into sub-groups, and the candidates in each sub-group are reordered based on the cost between the template samples and their corresponding prediction samples (i.e., reference samples). In this way, the candidates with better MVs (i.e., less template cost) are associated with smaller merge indices. Similarly, in MMVD mode, all possible MMVD refinement locations are reordered with the template cost, and only the top few locations after the reordering are allowed to be selected by the encoder / decoder. In this disclosure, methods of applying the proposed adaptive motion-compensated filtering to the cost calculation of the template matching based schemes are proposed.

[0286] In the first method, when the adaptive motion-compensated filtering is applied to one merge candidate, it is proposed to always bypass the adaptive motion-compensated filtering when calculating its template cost. However, if this candidate is selected (e.g., as indicated by the merge index), the adaptive motion-compensated filtering is still applied to generate the prediction samples of the block. To illustrate the above method, as shown in FIG. 2, assume there are L merge candidates, i.e., M0, M1, …, M FIG. 22 L-1, and the adaptive motion-compensated filtering is applied to M0and M L-1 ; in addition, without loss of generality, assume that the adaptive motion-compensated filtering is applied to the merge candidates M i and M j , while the adaptive motion-compensated filtering is not applied to the other merge candidates. With this method, during the template-based reordering process, the motion of the L merge candidates is always used to calculate the difference between the template samples and the corresponding template prediction samples. However, depending on whether M i or M j is finally selected, the adaptive motion-compensated filtering can still be applied to generate the final prediction samples of the block.

[0287] In the second method, it is proposed to apply adaptive motion compensation filtering to the calculation of the template cost and the generation of the block's prediction samples. As shown in FIG. 23 different from the first method, adaptive motion compensation filtering is applied to adjust the template prediction samples of M i and M j before their corresponding costs are calculated. Furthermore, during the reordering process, when a merge candidate is bi-predicted, different adaptive filtering methods can be applied to adjust the template's prediction samples. In one method (Method #1), it is proposed to use the method discussed in the section of "Adaptive bi-prediction filtering based on template bi-predicted samples" to generate the template prediction samples of each bi-predicted merge candidate. Specifically, in the proposed scheme, first, the bi-predicted samples of the template samples are generated according to the L0 and L1 MVs of the merge candidate; then, an adaptive filter is derived and applied to the bi-predicted template prediction samples, as shown in (14). In the second method (Method #2), it is proposed to use the method discussed in the section of "Adaptive bi-prediction filtering based on template uni-predicted samples" to generate the template prediction samples of each bi-predicted merge candidate. Specifically, in this method, first, the template prediction samples in L0 and L1 are generated using the MVs in L0 and L1, respectively; then, two adaptive filters are derived and applied to the template's L0 and L1 prediction samples in a one-sided manner, and then they are combined to generate the final prediction samples of the template, as shown in (18) and (19). In the third method (Method #3), it is proposed to use the method discussed in the section of "Adaptive bi-prediction filtering based on recursive uni-prediction filtering" to generate the template prediction samples of each bi-predicted merge candidate. Specifically, through this scheme, first, the template prediction samples in L0 and L1 are generated using the MVs in L0 and L1, respectively; then, two filters are derived iteratively and applied to the template's two uni-predicted samples, and then they are combined to generate the final prediction samples of the template. In practice, different adaptive filtering schemes can be applied to different template matching schemes, which can lead to different coding efficiency / complexity trade-offs. In one specific example, it is proposed to apply Method #3 to the ARMC mode and the regular MMVD mode, and to apply Method #2 to the affine MMVD mode.

[0288] FIG. 23 A computing environment (or computing device) 2310 is shown coupled with a user interface 2360. The computing environment 2310 can be part of a data processing server. In some embodiments, the computing device 2310 can perform any of a variety of methods or processes (such as encoding / decoding methods or processes) as described above according to various examples of the present disclosure. The computing environment 2310 can include a processor 2320, a memory 2340, and an I / O interface 2350.

[0289] The processor 2320 generally controls the overall operation of the computing environment 2310, such as the operations associated with displaying, data acquisition, data communication, and image processing. The processor 2320 can include one or more processors to execute instructions to perform all or some of the steps in the above methods. Additionally, the processor 2320 can include one or more modules that facilitate interaction between the processor 2320 and other components. The processor can be a central processing unit (CPU), a microprocessor, a microcomputer, a GPU, or the like.

[0290] The memory 2340 is configured to store various types of data to support the operation of the computing environment 2310. The memory 2340 can include predetermined software 2342. Examples of such data include instructions for any application programs or methods operating on the computing environment 2310, video data sets, image data, and the like. The memory 2340 can be implemented by using any type of volatile or non-volatile storage devices, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic or optical disks.

[0291] The I / O interface 2350 provides an interface between the processor 2320 and peripheral interface modules, such as a keyboard, a click wheel, a button, and the like. The buttons can include, but are not limited to, a home button, a start scanning button, and a stop scanning button. The I / O interface 2350 can be coupled with an encoder and a decoder.

[0292] In embodiments, a non-transitory computer readable storage medium is also provided that includes, for example, in the memory 2330, a plurality of programs executable by the processor 2320 in the computing environment 2310 for performing the above-described methods, and / or stores a bitstream or a data stream generated by the above-described encoding method or to be decoded by the above-described decoding method. In one example, the plurality of programs can be executed by the processor 2320 in the computing environment 2310 to receive (e.g., from a video encoder 20 in a video encoder 20) a bitstream or a data stream that includes encoded video information (e.g., video blocks representing encoded video frames and / or associated one or more syntax elements, etc.), and also can be executed by the processor 2320 in the computing environment 2310 to perform the above-described decoding method according to the received bitstream or data stream. In another example, the plurality of programs can be executed by the processor 2320 in the computing environment 2310 to perform the above-described encoding method to encode video information (e.g., video blocks representing video frames and / or associated one or more syntax elements, etc.) into a bitstream or a data stream, and also can be executed by the processor 2320 in the computing environment 2310 to transmit the bitstream or data stream (e.g., to a video decoder 30 in a video decoder 30). FIG. 1G In embodiments, a non-transitory computer readable storage medium is also provided that includes, for example, in the memory 2330, a plurality of programs executable by the processor 2320 in the computing environment 2310 for performing the above-described methods, and / or stores a bitstream or a data stream generated by the above-described encoding method or to be decoded by the above-described decoding method. In one example, the plurality of programs can be executed by the processor 2320 in the computing environment 2310 to receive (e.g., from a video encoder 20 in a video encoder 20) a bitstream or a data stream that includes encoded video information (e.g., video blocks representing encoded video frames and / or associated one or more syntax elements, etc.), and also can be executed by the processor 2320 in the computing environment 2310 to perform the above-described decoding method according to the received bitstream or data stream. In another example, the plurality of programs can be executed by the processor 2320 in the computing environment 2310 to perform the above-described encoding method to encode video information (e.g., video blocks representing video frames and / or associated one or more syntax elements, etc.) into a bitstream or a data stream, and also can be executed by the processor 2320 in the computing environment 2310 to transmit the bitstream or data stream (e.g., to a video decoder 30 in a video decoder 30).FIG. 2B The non-transitory computer-readable storage medium can store therein a bitstream or a data stream including encoded video information generated by an encoder (e.g., video encoder 20 in FIG. 2) using, for example, the encoding methods described above, for decoding by a decoder (e.g., video decoder 30 in FIG. 3). FIG. 1G The non-transitory computer-readable storage medium can store therein a bitstream or a data stream including encoded video information generated by an encoder (e.g., video encoder 20 in FIG. 2) using, for example, the encoding methods described above, for decoding by a decoder (e.g., video decoder 30 in FIG. 3). FIG. 2B The non-transitory computer-readable storage medium can store therein a bitstream or a data stream including encoded video information generated by an encoder (e.g., video encoder 20 in FIG. 2) using, for example, the encoding methods described above, for decoding by a decoder (e.g., video decoder 30 in FIG. 3).

[0293] In embodiments, a bitstream generated by the encoding methods described above or to be decoded by the decoding methods described above is provided. In embodiments, a bitstream including encoded video information generated by the encoding methods described above or to be decoded by the decoding methods described above is provided.

[0294] In embodiments, a computing device is also provided, including one or more processors (e.g., processor 2320) and a non-transitory computer-readable storage medium or memory 2330 having stored therein a plurality of programs executable by the one or more processors, wherein the one or more processors are configured to perform the above-described methods when executing the plurality of programs.

[0295] In embodiments, a computer program product is also provided, having instructions for storing or transmitting a bitstream including encoded video information generated by the encoding methods described above or to be decoded by the decoding methods described above. In embodiments, a computer program product is also provided, including a plurality of programs, for example in memory 2330, executable by a processor 2320 in a computing environment 2310 for performing the above-described methods. For example, the computer program product can include a non-transitory computer- readable storage medium.

[0296] In embodiments, the computing environment 2310 can be implemented with one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components, for performing the above-described methods.

[0297] In embodiments, a method of storing a bitstream is also provided, comprising storing the bitstream on a digital storage medium, wherein the bitstream comprises encoded video information generated by the above-mentioned encoding method or encoded video information to be decoded by the above-mentioned decoding method.

[0298] In embodiments, a method for transmitting a bitstream generated by the above-mentioned encoder is also provided. In embodiments, a method for receiving a bitstream to be decoded by the above-mentioned decoder is also provided.

[0299] FIG. 24 is a flowchart illustrating a video decoding method according to an example of the present disclosure. The method can be implemented by a decoder for decoding an inter-coded block. In step 2401, the method comprises obtaining, by the decoder, a target motion vector of the current inter-coded block from a candidate list based on a plurality of first reconstructed samples neighboring the current inter-coded block, wherein the candidate list comprises a plurality of motion vector candidates of the current inter-coded block. In step 2402, the method comprises obtaining, by the decoder, a plurality of first prediction samples based on the target motion vector of the current inter-coded block. In step 2403, the method comprises obtaining, by the decoder, a plurality of filtered prediction samples based on at least one template filter and the plurality of first prediction samples in response to determining that an adaptive motion compensation filtering is applied to the current inter-coded block, wherein the at least one template filter is obtained based on a current template of the current inter-coded block, wherein the current template comprises a plurality of second reconstructed samples neighboring the current inter-coded block.

[0300] In some examples, the at least one template filter is obtained by: obtaining, by the decoder, a current template of the current inter-coded block; obtaining, by the decoder, a plurality of template prediction samples of the current template based on the target motion vector; and obtaining, by the decoder, the at least one template filter based on the plurality of template prediction samples and the current template.

[0301] In some examples, obtaining the target motion vector of the current inter-coded block from the candidate list comprises: for each motion vector candidate, obtaining, by the decoder, a template matching cost of the plurality of first reconstructed samples and the plurality of second prediction samples corresponding to the plurality of first reconstructed samples; reordering, by the decoder, the candidate list based on the template matching cost of the plurality of motion vector candidates; and obtaining, based on the reordered candidate list, the target motion vector of the current inter-coded block.

[0302] In some examples, obtaining, by the decoder, the template matching cost for each motion vector candidate includes: obtaining, by the decoder, a plurality of second prediction samples of the plurality of first reconstructed samples based on the motion vector candidate; responsive to determining that adaptive motion compensation filtering is applied to the motion vector candidate, obtaining, by the decoder, a plurality of filtered template prediction samples based on at least one candidate filter and the plurality of second prediction samples; and obtaining the template matching cost based on the plurality of filtered template prediction samples and the plurality of first reconstructed samples, wherein the at least one candidate filter is obtained based on a plurality of third reconstructed samples neighboring the current inter-coded block and a plurality of third prediction samples corresponding to the third reconstructed samples.

[0303] In some examples, the at least one candidate filter is obtained by: obtaining, by the decoder, a plurality of third reconstructed samples of the current coded block; obtaining, by the decoder, a plurality of third prediction samples of the plurality of third reconstructed samples based on the motion vector candidate; and obtaining, by the decoder, the at least one candidate filter based on the plurality of third prediction samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples.

[0304] In some examples, obtaining, by the decoder, the at least one candidate filter based on the plurality of third prediction samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples includes: responsive to determining that bi-prediction filtering is applied to the motion vector candidate, obtaining a plurality of first combined prediction samples of the plurality of third reconstructed samples, each first combined prediction sample being obtained by combining two third prediction samples in different reference pictures; and obtaining coefficients of the one candidate filter by minimizing a difference between the plurality of combined prediction samples and the plurality of third reconstructed samples.

[0305] In some examples, obtaining, by the decoder, the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples includes: responsive to determining that bi-prediction filtering is applied to the motion vector candidate, obtaining, by the decoder, a plurality of second combined prediction samples of the plurality of first reconstructed samples based on the motion vector candidate, each second combined prediction sample being obtained by combining two second prediction samples in different reference pictures; and obtaining, by the decoder, the plurality of filtered template prediction samples by applying the candidate filter to the plurality of second combined prediction samples.

[0306] In some examples, obtaining, by the decoder, the plurality of third prediction samples of the plurality of third reconstructed samples based on the motion vector candidate includes: obtaining a first plurality of third prediction samples corresponding to a first motion vector candidate and a second plurality of third prediction samples corresponding to a second motion vector candidate.

[0307] In some examples, obtaining, by the decoder, the at least one candidate filter based on the plurality of third prediction samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples comprises: obtaining a first set of coefficients of a first candidate filter by minimizing a difference between a first plurality of third prediction samples using a L0 motion vector and the plurality of third reconstructed samples; and obtaining a second set of coefficients of a second candidate filter by minimizing a difference between a second plurality of third prediction samples using a L1 motion vector and the plurality of third reconstructed samples.

[0308] In some examples, obtaining, by the decoder, the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples comprises: obtaining a first plurality of filtered template prediction samples by applying the first candidate filter to the first plurality of second prediction samples; obtaining a second plurality of filtered template prediction samples by applying the second candidate filter to the second plurality of second prediction samples; and obtaining the plurality of filtered template prediction samples by combining the first plurality of filtered template prediction samples and the second plurality of filtered template prediction samples.

[0309] In some examples, the candidate list is applied to a motion vector difference (MMVD) mode of affine merge (AM) mode.

[0310] In some examples, obtaining, by the decoder, the at least one candidate filter based on the plurality of third prediction samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples comprises: calculating a plurality of target template samples based on the plurality of third reconstructed samples and a plurality of previously filtered prediction samples of the plurality of third reconstructed samples; obtaining coefficients of a current candidate filter by minimizing a difference between the plurality of current prediction samples of the plurality of third reconstructed samples and the plurality of target template samples; and calculating a plurality of current filtered prediction samples by applying the current candidate filter to the plurality of current prediction samples.

[0311] In some examples, obtaining, by the decoder, the at least one candidate filter based on the plurality of third prediction samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples further comprises: obtaining coefficients of a first candidate filter by minimizing a difference between the first plurality of third prediction samples and the plurality of third reconstructed samples; and calculating a plurality of previously filtered prediction samples by applying the first candidate filter to the first plurality of third prediction samples.

[0312] In some examples, obtaining, by the decoder, the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples comprises: obtaining a first plurality of filtered template prediction samples by applying a first candidate filter to a first plurality of second prediction samples; obtaining a second plurality of filtered template prediction samples by applying a second candidate filter to a second plurality of second prediction samples; and obtaining the plurality of filtered template prediction samples by combining the first plurality of filtered template prediction samples and the second plurality of filtered template prediction samples.

[0313] In some examples, the candidate list is applied to an adaptive reorder of merge candidates (ARMC) mode with template matching, or a regular merge mode (MMVD) mode with motion vector difference.

[0314] In some examples, the plurality of first reconstructed samples, the plurality of second reconstructed samples, and the plurality of third reconstructed samples are the same neighboring samples of the current inter coded block.

[0315] FIG. 25 is a flowchart illustrating a video encoding method according to an example of the present disclosure. The method can be implemented by an encoder for encoding an inter coded block. In step 2501, the method comprises obtaining, by the encoder, a target motion vector of the current inter coded block from a candidate list based on a plurality of first reconstructed samples neighboring the current inter coded block, wherein the candidate list comprises a plurality of motion vector candidates of the current inter coded block. In step 2502, the method comprises obtaining, by the encoder, a plurality of first prediction samples based on the target motion vector of the current inter coded block. In step 2503, the method comprises obtaining, by the encoder, a plurality of filtered prediction samples based on at least one template filter and the plurality of first prediction samples in response to determining that an adaptive motion compensated filtering is applied to the current inter coded block, wherein the at least one template filter is obtained based on a current template of the current inter coded block, wherein the current template comprises a plurality of second reconstructed samples neighboring the current inter coded block.

[0316] In some examples, the at least one template filter is obtained by: obtaining, by the encoder, a current template of the current inter coded block; obtaining, by the encoder, a plurality of template prediction samples of the current template based on the target motion vector; and obtaining, by the encoder, the at least one template filter based on the plurality of template prediction samples and the current template.

[0317] In some examples, obtaining, from the candidate list, the target motion vector of the current inter-coded block includes, for each motion vector candidate, obtaining, by the encoder, a template matching cost of the plurality of first reconstructed samples and the plurality of second predicted samples corresponding to the plurality of first reconstructed samples; reordering, by the encoder, the candidate list based on the template matching cost of the plurality of motion vector candidates; and obtaining, based on the reordered candidate list, the target motion vector of the current inter-coded block.

[0318] In some examples, obtaining, for each motion vector candidate, the template matching cost of the plurality of first reconstructed samples and the plurality of second predicted samples includes obtaining, by the encoder, the plurality of second predicted samples of the plurality of first reconstructed samples based on the motion vector candidate; in response to determining that the adaptive motion compensation filtering is applied to the motion vector candidate, obtaining, by the encoder, a plurality of filtered template predicted samples based on at least one candidate filter and the plurality of second predicted samples; and obtaining the template matching cost based on the plurality of filtered template predicted samples and the plurality of first reconstructed samples, wherein the at least one candidate filter is obtained based on a plurality of third reconstructed samples neighboring the current inter-coded block and a plurality of third predicted samples corresponding to the third reconstructed samples.

[0319] In some examples, the at least one candidate filter is obtained by: obtaining, by the encoder, the plurality of third reconstructed samples of the current coded block; obtaining, by the encoder, the plurality of third predicted samples of the plurality of third reconstructed samples based on the motion vector candidate; and obtaining, by the encoder, the at least one candidate filter based on the plurality of third predicted samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples.

[0320] In some examples, obtaining, by the encoder, the at least one candidate filter based on the plurality of third predicted samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples includes, in response to determining that the bi-prediction filtering is applied to the motion vector candidate, obtaining a plurality of first combined predicted samples of the plurality of third reconstructed samples, each first combined predicted sample being obtained by combining two third predicted samples in different reference pictures; and obtaining a coefficient of the one candidate filter by minimizing a difference between the plurality of combined predicted samples and the plurality of third reconstructed samples.

[0321] In some examples, obtaining, by the encoder, the plurality of filtered template predicted samples based on the at least one candidate filter and the plurality of second predicted samples includes, in response to determining that the bi-prediction filtering is applied to the motion vector candidate, obtaining, by the encoder, a plurality of second combined predicted samples of the plurality of first reconstructed samples based on the motion vector candidate, each second combined predicted sample being obtained by combining two second predicted samples in different reference pictures; and obtaining, by the encoder, the plurality of filtered template predicted samples by applying the candidate filter to the plurality of second combined predicted samples.

[0322] In some examples, obtaining, by the encoder, the plurality of third prediction samples of the plurality of third reconstructed samples based on the motion vector candidate comprises obtaining a first plurality of third prediction samples corresponding to the first motion vector candidate and a second plurality of third prediction samples corresponding to the second motion vector candidate.

[0323] In some examples, obtaining, by the encoder, the at least one candidate filter based on the plurality of third prediction samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples comprises obtaining a first set of coefficients of a first candidate filter by minimizing a difference between the first plurality of third prediction samples using the L0 motion vector and the plurality of third reconstructed samples, and obtaining a second set of coefficients of a second candidate filter by minimizing a difference between the second plurality of third prediction samples using the Ll motion vector and the plurality of third reconstructed samples.

[0324] In some examples, obtaining, by the encoder, the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples comprises obtaining a first plurality of filtered template prediction samples by applying the first candidate filter to the first plurality of second prediction samples, obtaining a second plurality of filtered template prediction samples by applying the second candidate filter to the second plurality of second prediction samples, and obtaining the plurality of filtered template prediction samples by combining the first plurality of filtered template prediction samples and the second plurality of filtered template prediction samples.

[0325] In some examples, the candidate list is applied to a motion vector difference (MMVD) mode of affine merge (AM) mode.

[0326] In some examples, obtaining, by the encoder, the at least one candidate filter based on the plurality of third prediction samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples comprises calculating a plurality of target template samples based on the plurality of third reconstructed samples and a plurality of previously filtered prediction samples of the plurality of third reconstructed samples, obtaining coefficients of a current candidate filter by minimizing a difference between a plurality of current prediction samples of the plurality of third reconstructed samples and the plurality of target template samples, and calculating a plurality of current filtered prediction samples by applying the current candidate filter to the plurality of current prediction samples.

[0327] In some examples, obtaining, by the encoder, the at least one candidate filter based on the plurality of third prediction samples of the plurality of third reconstructed samples and the plurality of third reconstructed samples further comprises obtaining coefficients of a first candidate filter by minimizing a difference between the first plurality of third prediction samples and the plurality of third reconstructed samples, and calculating a plurality of previously filtered prediction samples by applying the first candidate filter to the first plurality of third prediction samples.

[0328] In some examples, obtaining, by the encoder, the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples comprises: obtaining a first plurality of filtered template prediction samples by applying a first candidate filter to a first plurality of second prediction samples; obtaining a second plurality of filtered template prediction samples by applying a second candidate filter to a second plurality of second prediction samples; and obtaining the plurality of filtered template prediction samples by combining the first plurality of filtered template prediction samples and the second plurality of filtered template prediction samples.

[0329] In some examples, the candidate list is applied to an adaptive reorder of merge candidates (ARMC) mode with template matching, or a regular merge mode (MMVD) mode with motion vector difference.

[0330] In some examples, the plurality of first reconstructed samples, the plurality of second reconstructed samples, and the plurality of third reconstructed samples are the same neighboring samples of a current inter coded block.

[0331] In some examples, an apparatus for video coding is provided. The apparatus includes a processor 2320 and a memory 2340 configured to store instructions executable by the processor; wherein the processor, when executing the instructions, is configured to perform any of the methods as shown in FIG. 24 to FIG. 25

[0332] In some other examples, a non-transitory computer-readable storage medium having instructions stored therein is provided. When the instructions are executed by a processor 2320, the instructions cause the processor to perform any of the methods as shown in FIG. 19 to FIG. 21 In one example, a plurality of programs can be executed by the processor 2320 in the computing environment 2310 to receive (e.g., from a video encoder 20 in FIG. 1G In another example, a plurality of programs can be executed by the processor 2320 in the computing environment 2310 to perform the encoding methods described above to encode video information (e.g., video blocks representing video frames and / or associated one or more syntax elements, etc.) into a bitstream or data stream, and also can be executed by the processor 2320 in the computing environment 2310 to transmit the bitstream or data stream (e.g., to a video decoder 30 in FIG. 2B Alternatively, the non-transitory computer-readable storage medium can have a bitstream or data stream stored therein, which includes video information (e.g., video blocks representing video frames and / or associated one or more syntax elements, etc.) encoded by an encoder (e.g., a video encoder 20 in FIG. 1G ​encoded video information (e.g., video blocks representing encoded video frames and / or associated one or more syntax elements, etc.) generated using, for example, the encoding methods described above, for decoding by a decoder (e.g., video decoder 30 in FIG. 1) in a later time. FIG. 2B The video data is decoded by a video decoder (e.g., video decoder 30 in FIG. 1) using the encoded video information (e.g., video blocks representing encoded video frames and / or associated one or more syntax elements, etc.) generated using, for example, the encoding methods described above, for decoding by a decoder (e.g., video decoder 30 in FIG. 1) in a later time.

[0333] The description of the present disclosure is presented for purposes of illustration and is not intended to exhaustively delineate or limit the disclosure. Many modifications, variations, and alterations to the present disclosure will become apparent to those of ordinary skill in the art, once the teachings of the present disclosure have been made known.

[0334] Unless specifically stated otherwise, the order of steps in the methods according to the present disclosure is not intended to be limiting, and the steps in the methods according to the present disclosure can be altered, combined, or omitted according to actual circumstances. Also, at least one step in the methods according to the present disclosure can be adjusted, combined, or omitted according to actual needs.

[0335] The examples are chosen and described in order to explain the principles of the present disclosure and to enable others skilled in the art to best utilize the various embodiments of the present disclosure, and to best utilize the various embodiments for the intended purposes. Therefore, the scope of the present disclosure is not intended to be limited to the specific examples disclosed, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

[0336] The above-described methods can be implemented using an apparatus including one or more circuits, such as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components. The apparatus can use these circuits in combination with other hardware or software components to perform the above-described methods. Each of the modules, sub-modules, units, or sub-units disclosed above can be implemented using the one or more circuits at least in part.

[0337] Other examples of the disclosure, in addition to those shown and described herein, will become apparent to those skilled in the art from a consideration of the specification and practice of the disclosure. The application is intended to cover any variations, uses, or adaptations of the disclosure including those that are within its generic principles, field of use, or the like. The specification and examples are to be considered exemplary only.

[0338] It should be understood that the present disclosure is not limited to the precise examples described and shown in the accompanying drawings, and that various modifications and changes can be made without departing from its scope.

Claims

1. A video decoding method, comprising: The decoder obtains the target motion vector of the current inter-frame coding block from a candidate list based on multiple first reconstructed samples adjacent to the current inter-frame coding block, wherein the candidate list includes multiple motion vector candidates of the current inter-frame coding block; The decoder obtains a plurality of first prediction samples based on the target motion vector of the current inter-frame coding block; and In response to determining that adaptive motion compensation filtering is applied to the current inter-frame coded block, the decoder obtains a plurality of filtered prediction samples based on at least one template filter and the plurality of first prediction samples. The at least one template filter is obtained based on the current template of the current inter-frame coding block, wherein the current template includes a plurality of second reconstructed samples adjacent to the current inter-frame coding block.

2. The method as described in claim 1, wherein, The at least one template filter is obtained through the following steps: The current template of the current inter-frame coded block is obtained by the decoder; The decoder obtains multiple template prediction samples of the current template based on the target motion vector; as well as The decoder obtains the at least one template filter based on the multiple template prediction samples and the current template.

3. The method as described in claim 1, wherein, Obtaining the target motion vector of the current inter-frame coded block from the candidate list includes: For each motion vector candidate, obtain the template matching cost of the plurality of first reconstructed samples and the plurality of second predicted samples corresponding to the plurality of first reconstructed samples; The candidate list is reordered based on the template matching cost of the multiple motion vector candidates; and The target motion vector of the current inter-frame coding block is obtained based on the reordered candidate list.

4. The method of claim 3, wherein, For each motion vector candidate, the template matching cost for obtaining the plurality of first reconstructed samples and the plurality of second predicted samples includes: Based on motion vector candidates, the plurality of second predicted samples of the plurality of first reconstructed samples are obtained; In response to determining that an adaptive motion compensation filter is applied to the motion vector candidate, a plurality of filtered template prediction samples are obtained based on at least one candidate filter and the plurality of second prediction samples; and The template matching cost is obtained based on the multiple filtered template prediction samples and the multiple first reconstructed samples. The at least one candidate filter is obtained based on a plurality of third reconstruction samples adjacent to the current inter-frame coding block and a plurality of third prediction samples corresponding to the third reconstruction samples.

5. The method of claim 4, wherein, The at least one candidate filter is obtained through the following steps: The decoder obtains the plurality of third reconstructed samples of the current coded block; The decoder obtains multiple third predicted samples of the multiple third reconstructed samples based on the motion vector candidates; as well as The decoder obtains the at least one candidate filter based on the plurality of third prediction samples and the plurality of third reconstruction samples.

6. The method of claim 5, wherein, Obtaining the at least one candidate filter based on the plurality of third predicted samples and the plurality of third reconstructed samples includes: In response to determining that a bidirectional prediction filter is applied to motion vector candidates, a plurality of first combined prediction samples of the plurality of third reconstructed samples are obtained, each first combined prediction sample being obtained by combining two third prediction samples from different reference images; and The coefficients of a candidate filter are obtained by minimizing the difference between the plurality of combined predicted samples and the plurality of third reconstructed samples.

7. The method of claim 6, wherein, The decoder obtains the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples: In response to determining that bidirectional prediction filtering is applied to motion vector candidates, the decoder obtains a plurality of second combined prediction samples based on the plurality of first reconstructed samples, each second combined prediction sample being obtained by combining two second prediction samples from different reference images; as well as The decoder obtains the plurality of filtered template prediction samples by applying the candidate filter to the plurality of second combined prediction samples.

8. The method of claim 5, wherein, The plurality of third predicted samples obtained based on the motion vector candidates include: Obtain a first plurality of third prediction samples corresponding to the first motion vector candidate and a second plurality of third prediction samples corresponding to the second motion vector candidate.

9. The method of claim 8, wherein, Obtaining the at least one candidate filter based on the plurality of third predicted samples and the plurality of third reconstructed samples includes: A first set of coefficients for the first candidate filter is obtained by minimizing the difference between the first plurality of third predicted samples using the L0 motion vector and the plurality of third reconstructed samples; and The second set of coefficients for the second candidate filter is obtained by minimizing the difference between the second plurality of third predicted samples using the L1 motion vector and the plurality of third reconstructed samples.

10. The method of claim 9, wherein, Obtaining the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples includes: The first plurality of filtered template prediction samples are obtained by applying the first candidate filter to the first plurality of second prediction samples; A second plurality of filtered template prediction samples are obtained by applying the second candidate filter to a second plurality of second prediction samples; and The plurality of filtered template prediction samples are obtained by combining the first plurality of filtered template prediction samples and the second plurality of filtered template prediction samples.

11. The method of claim 10, wherein, The candidate list is applied to the affine merging mode (MMVD) mode that utilizes motion vector differences.

12. The method of claim 8, wherein, Obtaining the at least one candidate filter based on the plurality of third predicted samples and the plurality of third reconstructed samples includes: Multiple target template samples are calculated based on the multiple third reconstructed samples and the multiple previously filtered predicted samples of the multiple third reconstructed samples; The coefficients of the current candidate filter are obtained by minimizing the differences between the multiple currently predicted samples of the multiple third reconstructed samples and the multiple target template samples; and Multiple currently filtered prediction samples are calculated by applying the current candidate filter to the multiple currently predicted samples.

13. The method of claim 12, wherein, Obtaining the at least one candidate filter based on the plurality of third predicted samples and the plurality of third reconstructed samples further includes: The coefficients of the first candidate filter are obtained by minimizing the difference between the first plurality of third predicted samples and the plurality of third reconstructed samples; and The plurality of previously filtered prediction samples are calculated by applying the first candidate filter to the first plurality of third prediction samples.

14. The method of claim 12, wherein, Obtaining the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples includes: A first plurality of filtered template prediction samples are obtained by applying a first candidate filter to a first plurality of second prediction samples; A second plurality of filtered template prediction samples are obtained by applying a second candidate filter to a second plurality of second prediction samples; and The plurality of filtered template prediction samples are obtained by combining the first plurality of filtered template prediction samples and the second plurality of filtered template prediction samples.

15. The method of claim 14, wherein, The candidate list is applied to either the Merge Candidate Adaptive Reordering (ARMC) mode which utilizes template matching, or the conventional Merge Mode (MMVD) mode which utilizes motion vector difference.

16. The method of claim 4, wherein, The plurality of first reconstructed samples, the plurality of second reconstructed samples, and the plurality of third reconstructed samples are the same neighboring samples of the current inter-frame coding block.

17. A video coding method, comprising: The encoder obtains the target motion vector of the current inter-frame coding block from a candidate list based on multiple first reconstructed samples adjacent to the current inter-frame coding block, wherein the candidate list includes multiple motion vector candidates of the current inter-frame coding block; The encoder obtains a plurality of first prediction samples based on the target motion vector of the current inter-frame coding block; and In response to determining that adaptive motion compensation filtering is applied to the current inter-frame coding block, the encoder obtains a plurality of filtered prediction samples based on at least one template filter and the plurality of first prediction samples. The at least one template filter is obtained based on the current template of the current inter-frame coding block, wherein the current template includes a plurality of second reconstructed samples adjacent to the current inter-frame coding block.

18. The method of claim 17, wherein, The at least one template filter is obtained through the following steps: The encoder obtains the current template of the current inter-frame coded block; The encoder obtains multiple template prediction samples of the current template based on the target motion vector; as well as The encoder obtains at least one template filter based on the multiple template prediction samples and the current template.

19. The method of claim 17, wherein, Obtaining the target motion vector of the current inter-frame coded block from the candidate list includes: For each motion vector candidate, obtain the template matching cost of the plurality of first reconstructed samples and the plurality of second predicted samples corresponding to the plurality of first reconstructed samples; The candidate list is reordered based on the template matching cost of the multiple motion vector candidates; and The target motion vector of the current inter-frame coding block is obtained based on the reordered candidate list.

20. The method of claim 19, wherein, For each motion vector candidate, the template matching cost for obtaining the plurality of first reconstructed samples and the plurality of second predicted samples includes: Based on motion vector candidates, the plurality of second predicted samples of the plurality of first reconstructed samples are obtained; In response to determining that an adaptive motion compensation filter is applied to the motion vector candidate, a plurality of filtered template prediction samples are obtained based on at least one candidate filter and the plurality of second prediction samples; and The template matching cost is obtained based on the multiple filtered template prediction samples and the multiple first reconstructed samples. The at least one candidate filter is obtained based on a plurality of third reconstruction samples adjacent to the current inter-frame coding block and a plurality of third prediction samples corresponding to the third reconstruction samples.

21. The method of claim 20, wherein, The at least one candidate filter is obtained through the following steps: The encoder obtains the plurality of third reconstructed samples of the current coding block; The encoder obtains multiple third predicted samples of the multiple third reconstructed samples based on the motion vector candidates; as well as The encoder obtains the at least one candidate filter based on the plurality of third prediction samples and the plurality of third reconstruction samples.

22. The method of claim 21, wherein, Obtaining the at least one candidate filter based on the plurality of third predicted samples and the plurality of third reconstructed samples includes: In response to determining that a bidirectional prediction filter is applied to motion vector candidates, a plurality of first combined prediction samples of the plurality of third reconstructed samples are obtained, each first combined prediction sample being obtained by combining two third prediction samples from different reference images; and The coefficients of a candidate filter are obtained by minimizing the difference between the plurality of combined predicted samples and the plurality of third reconstructed samples.

23. The method of claim 22, wherein, The encoder obtains the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples: In response to determining that bidirectional prediction filtering is applied to motion vector candidates, the encoder obtains a plurality of second combined prediction samples of the plurality of first reconstructed samples based on the motion vector candidates, each second combined prediction sample being obtained by combining two second prediction samples from different reference images; as well as The encoder obtains the plurality of filtered template prediction samples by applying the candidate filter to the plurality of second combined prediction samples.

24. The method of claim 21, wherein, The plurality of third predicted samples obtained based on the motion vector candidates include: Obtain a first plurality of third prediction samples corresponding to the first motion vector candidate and a second plurality of third prediction samples corresponding to the second motion vector candidate.

25. The method of claim 24, wherein, Obtaining the at least one candidate filter based on the plurality of third predicted samples and the plurality of third reconstructed samples includes: A first set of coefficients for the first candidate filter is obtained by minimizing the difference between the first plurality of third predicted samples using the L0 motion vector and the plurality of third reconstructed samples; and The second set of coefficients for the second candidate filter is obtained by minimizing the difference between the second plurality of third predicted samples using the L1 motion vector and the plurality of third reconstructed samples.

26. The method of claim 25, wherein, Obtaining the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples includes: The first plurality of filtered template prediction samples are obtained by applying the first candidate filter to the first plurality of second prediction samples; A second plurality of filtered template prediction samples are obtained by applying the second candidate filter to a second plurality of second prediction samples; and The plurality of filtered template prediction samples are obtained by combining the first plurality of filtered template prediction samples and the second plurality of filtered template prediction samples.

27. The method of claim 26, wherein, The candidate list is applied to the affine merging mode (MMVD) mode that utilizes motion vector differences.

28. The method of claim 24, wherein, Obtaining the at least one candidate filter based on the plurality of third predicted samples and the plurality of third reconstructed samples includes: Multiple target template samples are calculated based on the multiple third reconstructed samples and the multiple previously filtered predicted samples of the multiple third reconstructed samples; The coefficients of the current candidate filter are obtained by minimizing the differences between the multiple currently predicted samples of the multiple third reconstructed samples and the multiple target template samples; and Multiple currently filtered prediction samples are calculated by applying the current candidate filter to the multiple currently predicted samples.

29. The method of claim 28, wherein, Obtaining the at least one candidate filter based on the plurality of third predicted samples and the plurality of third reconstructed samples further includes: The coefficients of the first candidate filter are obtained by minimizing the difference between the first plurality of third predicted samples and the plurality of third reconstructed samples; and The plurality of previously filtered prediction samples are calculated by applying the first candidate filter to the first plurality of third prediction samples.

30. The method of claim 28, wherein, Obtaining the plurality of filtered template prediction samples based on the at least one candidate filter and the plurality of second prediction samples includes: A first plurality of filtered template prediction samples are obtained by applying a first candidate filter to a first plurality of second prediction samples; A second plurality of filtered template prediction samples are obtained by applying a second candidate filter to a second plurality of second prediction samples; and The plurality of filtered template prediction samples are obtained by combining the first plurality of filtered template prediction samples and the second plurality of filtered template prediction samples.

31. The method of claim 30, wherein, The candidate list is applied to either the Merge Candidate Adaptive Reordering (ARMC) mode which utilizes template matching, or the conventional Merge Mode (MMVD) mode which utilizes motion vector difference.

32. The method of claim 20, wherein, The plurality of first reconstructed samples, the plurality of second reconstructed samples, and the plurality of third reconstructed samples are the same neighboring samples of the current inter-frame coding block.

33. An apparatus for video decoding, the apparatus comprising: One or more processors; as well as A memory coupled to the one or more processors and configured to store instructions executable by the one or more processors. The one or more processors are configured to perform the method as described in any one of claims 1 to 16 when executing the instructions.

34. A non-transitory computer-readable storage medium for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to perform the method as described in any one of claims 1 to 16.

35. An apparatus for video encoding, the apparatus comprising: One or more processors; as well as A memory coupled to the one or more processors and configured to store instructions executable by the one or more processors. The one or more processors are configured to perform the method as described in any one of claims 17 to 32 when executing the instructions.

36. A non-transitory computer-readable storage medium for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to perform the method as described in any one of claims 17 to 32.

37. A non-transitory computer-readable storage medium for storing a bit stream to be decoded by the method of any one of claims 1-16.

38. A non-transitory computer-readable storage medium for storing a bit stream generated by the method of any one of claims 17-32.

Images