Method and apparatus for adaptive motion compensation filtering

Abstract

Methods, apparatus, and non-transitory computer-readable storage media for video decoding and encoding are provided. In a method for video decoding, a decoder may: determine a non-adjacent neighboring block of a current inter-coded block, where the non-adjacent neighboring block includes a plurality of reconstruction sample points that are non-adjacent to the current inter-coded block; obtaining a plurality of prediction sample points of the non-adjacent adjacent blocks based on the motion vectors of the non-adjacent adjacent blocks; obtaining a filter based on the plurality of prediction sample points and the plurality of reconstruction sample points; obtaining a current prediction block based on the motion vector of the current inter-frame coding block; and obtaining a filtered prediction block based on the filter and the current prediction block.

Description

Method and apparatus for adaptive motion compensation filtering

[0001] Cross-Reference to Related Applications This application is filed and claims priority to U.S. Provisional Application No. 63 / 542,518, filed October 4, 2023, and U.S. Provisional Application No. 63 / 609,272, filed December 12, 2023, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0002] This disclosure relates to video encoding / decoding and compression, and more specifically, but not limited to, methods and apparatus for improving the encoding / decoding efficiency of inter-frame coded blocks. Background Technology

[0003] Various video codec techniques can be used to compress video data. Video codecs are performed according to one or more video codec standards. For example, video codec standards include Universal Video Codec (VVC), High Efficiency Video Codec (H.265 / HEVC), High-Advanced Video Codec (H.264 / AVC), and Moving Picture Experts Group (MPEG) codecs. Video codecs typically utilize predictive methods (e.g., inter-frame prediction, intra-frame prediction), which take advantage of redundancy present in video images or sequences. A key goal of video codec techniques is to compress video data to a lower bitrate while avoiding or minimizing video quality degradation.

[0004] The first version of the VVC standard was completed in July 2020, offering approximately 50% bitrate savings or equivalent perceived quality compared to its predecessor, HEVC. While VVC provides significant codec improvements over its predecessor, there is evidence that superior codec efficiency can be achieved using additional codec tools. Recently, the Joint Video Exploration Group (JVET), in collaboration with ITU-TVECG and ISO / IEC MPEG, began exploring advanced technologies that could achieve significantly higher codec efficiency than VVC. In April 2021, a software codebase called the Enhanced Compression Model (ECM) was established for future video codec exploration. The ECM reference software is based on the VVC Test Model (VTM) developed by JVET for VVC, and several existing modules (e.g., intra / inter-frame prediction, transform, loop filters, etc.) have been further extended and / or improved. In the future, any new codec tools beyond the VVC standard will need to be integrated into the ECM platform and tested using the JVET Common Test Conditions (CTC). Summary of the Invention

[0005] This disclosure provides examples of techniques related to improving the encoding / decoding efficiency of inter-frame coded blocks.

[0006] According to a first aspect of this disclosure, a method for video decoding of inter-frame coded blocks is provided. The method includes: determining, by a decoder, non-adjacent neighbor blocks of a current inter-frame coded block, wherein the non-adjacent neighbor blocks include a plurality of reconstructed samples that are non-adjacent to the current inter-frame coded block; obtaining, by the decoder, a plurality of predicted samples of the non-adjacent neighbor blocks based on motion vectors of the non-adjacent neighbor blocks; obtaining, by the decoder, a filter based on the plurality of predicted samples and the plurality of reconstructed samples; obtaining, by the decoder, a current prediction block based on motion vectors of the current inter-frame coded block; and obtaining a filtered prediction block based on the decoder, the filter, and the current prediction block.

[0007] According to a second aspect of this disclosure, a method for video decoding of inter-frame coded blocks is provided. The method includes: obtaining, by a decoder, filter coefficients of a filter applied to a previous decoded block of the current inter-frame coded block; obtaining, by the decoder, a current prediction block based on the motion vector of the current inter-frame coded block; and obtaining, by the decoder, a filtered prediction block based on the filter and the current prediction block; wherein the filter is derived based on a plurality of predicted samples from non-adjacent neighbor blocks of the previous decoded block, and the non-adjacent neighbor blocks, the non-adjacent neighbor blocks comprising a plurality of reconstructed samples that are not adjacent to the previous decoded block.

[0008] According to a third aspect of this disclosure, a method for video decoding of inter-frame coded blocks is provided. The method includes: obtaining a filter using a decoder based on a plurality of reconstructed samples of a current inter-frame coded block and a plurality of predicted samples of the plurality of reconstructed samples; obtaining a current predicted block using the decoder based on a motion vector of the current inter-frame coded block; and obtaining a filtered predicted block using the decoder based on the filter and the current predicted block, wherein the plurality of reconstructed samples are located in an upper region or a left region adjacent to the current inter-frame coded block.

[0009] According to a fourth aspect of this disclosure, a method for video decoding of inter-frame coded blocks is provided. The method includes: determining, by a decoder, adjacent or non-adjacent neighbor blocks of a current inter-frame coded block, wherein the adjacent or non-adjacent neighbor blocks include a plurality of reconstructed samples adjacent or non-adjacent to the current inter-frame coded block; obtaining, by the decoder, a plurality of predicted samples of a portion of the plurality of reconstructed samples; obtaining, by the decoder, a filter based on the plurality of predicted samples and the portion of the plurality of reconstructed samples; obtaining, by the decoder, a current predicted block based on the motion vector of the current inter-frame coded block; and obtaining, by the decoder, a filtered predicted block based on the filter and the current predicted block.

[0010] According to a fifth aspect of this disclosure, a method for video decoding of inter-frame coded blocks is provided. The method includes: determining, by a decoder, adjacent or non-adjacent neighbor blocks of a current inter-frame coded block, wherein the adjacent or non-adjacent neighbor blocks comprise a plurality of reconstructed samples adjacent or non-adjacent to the current inter-frame coded block; obtaining, by the decoder, a plurality of predicted samples of the plurality of reconstructed samples; obtaining, by the decoder, a filter based on the plurality of predicted samples and the plurality of reconstructed samples, wherein the coefficients of the filter include an offset; calculating, by the decoder, an offset adjustment for the offset based on the coefficients of the filter, the plurality of predicted samples, the plurality of reconstructed samples, and the total number of samples in the adjacent or non-adjacent neighbor blocks; adjusting, by the decoder, the coefficients of the filter based on the offset adjustment; obtaining, by the decoder, a current prediction block based on the motion vector of the current inter-frame coded block; and obtaining, by the decoder, a filtered prediction block based on the filter adjusted based on the offset adjustment and the current prediction block.

[0011] According to a sixth aspect of this disclosure, a method for video coding of inter-frame coded blocks is provided. The method includes: determining, by an encoder, non-adjacent neighbor blocks of a current inter-frame coded block, wherein the non-adjacent neighbor blocks include a plurality of reconstructed samples that are non-adjacent to the current inter-frame coded block; obtaining, by the encoder, a plurality of predicted samples of the non-adjacent neighbor blocks based on motion vectors of the non-adjacent neighbor blocks; obtaining, by the encoder, a filter based on the plurality of predicted samples and the plurality of reconstructed samples; obtaining, by the encoder, a current prediction block based on motion vectors of the current inter-frame coded block; and obtaining a filtered prediction block based on the encoder, the filter, and the current prediction block.

[0012] According to a seventh aspect of this disclosure, a method for video coding of inter-frame coded blocks is provided. The method includes: obtaining, by an encoder, filter coefficients of a filter applied to a previous coded block of the current inter-frame coded block; obtaining, by the encoder, a current prediction block based on a motion vector of the current inter-frame coded block; and obtaining, by the encoder, a filtered prediction block based on the filter and the current prediction block; wherein the filter is derived based on a plurality of predicted samples from non-adjacent neighbor blocks of the previous coded block, and the non-adjacent neighbor blocks comprising a plurality of reconstructed samples that are not adjacent to the previous coded block.

[0013] According to an eighth aspect of this disclosure, a method for video coding of inter-frame coded blocks is provided. The method includes: obtaining a filter by an encoder based on a plurality of reconstructed samples of a current inter-frame coded block and a plurality of predicted samples of the plurality of reconstructed samples; obtaining a current predicted block by the encoder based on a motion vector of the current inter-frame coded block; and obtaining a filtered predicted block by the encoder based on the filter and the current predicted block, wherein the plurality of reconstructed samples are located in an upper region or a left region adjacent to the current inter-frame coded block.

[0014] According to a ninth aspect of this disclosure, a method for video coding of inter-frame coded blocks is provided. The method includes: determining, by an encoder, adjacent or non-adjacent neighbor blocks of a current inter-frame coded block, wherein the adjacent or non-adjacent neighbor blocks include a plurality of reconstructed samples adjacent or non-adjacent to the current inter-frame coded block; obtaining, by the encoder, a plurality of predicted samples of a portion of the plurality of reconstructed samples; obtaining, by the encoder, a filter based on the plurality of predicted samples and the portion of the plurality of reconstructed samples; obtaining, by the encoder, a current predicted block based on the motion vector of the current inter-frame coded block; and obtaining, by the encoder, a filtered predicted block based on the filter and the current predicted block.

[0015] According to a tenth aspect of this disclosure, a method for video coding of inter-frame coded blocks is provided. The method includes: determining, by an encoder, adjacent or non-adjacent neighbor blocks of a current inter-frame coded block, wherein the adjacent or non-adjacent neighbor blocks comprise a plurality of reconstructed samples adjacent or non-adjacent to the current inter-frame coded block; obtaining, by the encoder, a plurality of predicted samples of the plurality of reconstructed samples; obtaining, by the encoder, a filter based on the plurality of predicted samples and the plurality of reconstructed samples, wherein the coefficients of the filter include an offset; calculating, by the encoder, an offset adjustment for the offset based on the coefficients of the filter, the plurality of predicted samples, the plurality of reconstructed samples, and the total number of samples in the adjacent or non-adjacent neighbor blocks; adjusting, by the encoder, the coefficients of the filter based on the offset adjustment; obtaining, by the encoder, a current prediction block based on the motion vector of the current inter-frame coded block; and obtaining, by the encoder, a filtered prediction block based on the filter adjusted based on the offset adjustment and the current prediction block.

[0016] According to the eleventh aspect of this disclosure, an apparatus for video decoding is provided. The apparatus may include: one or more processors; and a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors. Furthermore, the one or more processors, when executing the instructions, are configured to perform the method according to the first, second, third, fourth, or fifth aspect.

[0017] According to a twelfth aspect of this disclosure, an apparatus for video encoding is provided. The apparatus may include: one or more processors; and a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors. Furthermore, the one or more processors, when executing the instructions, are configured to perform the method according to a sixth, seventh, eighth, ninth, or tenth aspect.

[0018] According to a thirteenth aspect of this disclosure, a non-transitory computer-readable storage medium is provided for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to perform the method according to the first aspect, second aspect, third aspect, fourth aspect, or fifth aspect.

[0019] According to the fourteenth aspect of this disclosure, a non-transitory computer-readable storage medium is provided for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to perform the method according to the sixth, seventh, eighth, ninth, or tenth aspect.

[0020] According to the fifteenth aspect of this disclosure, a non-transitory computer-readable storage medium is provided for storing a bit stream that will be decoded by the method described in accordance with the first, second, third, fourth, or fifth aspect.

[0021] According to the sixteenth aspect of this disclosure, a non-transitory computer-readable storage medium is provided for storing a bit stream generated by the method described in accordance with the sixth, seventh, eighth, ninth, or tenth aspect. Attached Figure Description

[0022] A more specific description of the examples of this disclosure will be presented with reference to the specific examples shown in the accompanying drawings. Given that these drawings depict only a few examples and are therefore not intended to limit the scope, the examples will be described and explained with additional features and details using the drawings.

[0023] Figure 1A is a block diagram illustrating a system for encoding and decoding video blocks according to some examples of this disclosure.

[0024] Figure 1B is a block diagram of an encoder according to some examples of this disclosure.

[0025] Figures 1C to 1F are block diagrams illustrating how, according to some examples of this disclosure, a frame is recursively divided into multiple video blocks of different sizes and shapes.

[0026] Figure 1G is a block diagram of an exemplary video encoder according to some examples of this disclosure.

[0027] Figure 2A is a block diagram of a decoder according to some examples of this disclosure.

[0028] Figure 2B is a block diagram illustrating an exemplary video decoder according to some examples of this disclosure.

[0029] Figure 3A is a diagram illustrating block segmentation in a multi-type tree structure according to some examples of this disclosure.

[0030] Figure 3B is a diagram illustrating block segmentation in a multi-type tree structure according to some examples of this disclosure.

[0031] Figure 3C is a diagram illustrating block segmentation in a multi-type tree structure according to some examples of this disclosure.

[0032] Figure 3D is a diagram illustrating block segmentation in a multi-type tree structure according to some examples of this disclosure.

[0033] Figure 3E is a diagram illustrating block segmentation in a multi-type tree structure according to some examples of this disclosure.

[0034] Figure 4 illustrates some examples of d according to this disclosure. x and d y This is an example of the horizontal and vertical values ​​of MV.

[0035] Figure 5 illustrates an example of an MV having a fractional value and an interpolation filter applied to generate corresponding predicted samples at the fractional sample locations according to some examples of the present invention.

[0036] Figure 6 shows examples of two diamond filter shapes according to some examples of this disclosure.

[0037] Figure 7 illustrates a subsampled 1-D Laplace calculation of gradients applied to all directions according to some examples of this disclosure.

[0038] Figure 8 is a diagram illustrating local illumination compensation (LIC) for unidirectional prediction according to some examples of this disclosure.

[0039] Figures 9A and 9B are diagrams illustrating the generation of LIC template prediction samples for affine patterns according to some examples of this disclosure.

[0040] Figure 10 is a block diagram of video coding with adaptive filtering for bidirectional prediction according to some examples of this disclosure.

[0041] Figure 11 is a block diagram of video decoding with adaptive filtering for bidirectional prediction according to some examples of this disclosure.

[0042] Figure 12 is a diagram illustrating adaptive motion compensation filtering for template-based bidirectional predictive samples according to some examples of this disclosure.

[0043] Figure 13 is a diagram illustrating adaptive motion compensation filtering for template-based unidirectional predictive samples according to some examples of this disclosure.

[0044] Figure 14 is a diagram illustrating an OBMC process for encoding and decoding a CU without sub-block motion compensation, according to some examples of this disclosure.

[0045] Figure 15 is a diagram illustrating the OBMC process for a CU encoded and decoded via sub-block mode according to some examples of this disclosure.

[0046] Figure 16 is a diagram illustrating a template-based OBMC according to some examples of this disclosure.

[0047] Figures 17A and 17B are diagrams illustrating non-adjacent neighbor blocks of different sizes according to some examples of this disclosure.

[0048] Figure 18 is a diagram showing a template for cost calculation in non-sub-block merging mode in ARMC and its corresponding reference samples, according to some examples of this disclosure.

[0049] Figure 19 is a diagram showing a template for cost calculation of a sub-block merging mode in ARMC according to some examples of this disclosure, and its corresponding reference samples.

[0050] Figure 20 is a diagram showing the refined positions along a k×π / 8 diagonal angle (reflected by points with three different patterns) around a basic candidate (reflected by the central star) according to some examples of this disclosure.

[0051] Figure 21 is a diagram illustrating, according to some examples of this disclosure, the application of adaptive MC filtering when generating prediction samples for the current block, but bypassing the adaptive MC filtering when calculating the template cost of different merge candidates / MMVD candidates.

[0052] Figure 22 is a diagram illustrating the application of adaptive MC filtering in generating prediction samples for the current block and calculating template costs for different merge candidates / MMVD candidates, according to some examples of this disclosure.

[0053] Figure 23 is a diagram illustrating the selection of merge candidates for the AMVP merge mode when at least one merge candidate associated with adaptive motion compensation filtering is present, according to some examples of this disclosure.

[0054] Figure 24 is a graph showing selected samples of some examples used to derive adaptive filter coefficients for adjacent / non-adjacent inter-frame CUs according to this disclosure.

[0055] Figure 25 is a diagram illustrating a computing environment coupled with a user interface according to some examples of this disclosure.

[0056] Figure 26 is a flowchart illustrating a method for video decoding according to some examples of this disclosure.

[0057] Figure 27 is a flowchart illustrating a method for video decoding according to some examples of this disclosure.

[0058] Figure 28 is a flowchart illustrating a method for video decoding according to some examples of this disclosure.

[0059] Figure 29 is a flowchart illustrating a method for video decoding according to some examples of this disclosure.

[0060] Figure 30 is a flowchart illustrating a method for video decoding according to some examples of this disclosure.

[0061] Figure 31 is a flowchart illustrating a method for video encoding according to some examples of this disclosure.

[0062] Figure 32 is a flowchart illustrating a method for video encoding according to some examples of this disclosure.

[0063] Figure 33 is a flowchart illustrating some examples of methods for video encoding according to this disclosure.

[0064] Figure 34 is a flowchart illustrating some examples of methods for video encoding according to this disclosure.

[0065] Figure 35 is a flowchart illustrating some examples of methods for video encoding according to this disclosure. Detailed Implementation

[0066] Referring now to the detailed description, examples of which are illustrated in the accompanying drawings. Numerous non-limiting specific details are set forth in the following detailed description to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives can be used. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented on many types of electronic devices with digital video capabilities.

[0067] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. Unless otherwise expressly indicated throughout the disclosure, the singular forms “a / an,” “the,” and “the” used in this disclosure and the appended claims are also intended to include the plural forms. It should also be understood that the term “and / or” as used in this disclosure refers to and includes one or any of the plurality of related items listed or all possible combinations thereof.

[0068] Throughout this specification, references to "an embodiment," "an embodiment," "an example," "some embodiments," "some examples," or similar language mean that a particular feature, structure, or characteristic described is included in at least one embodiment or example. Unless otherwise expressly stated, the features, structures, elements, or characteristics described in connection with one or more embodiments also apply to other embodiments.

[0069] Throughout this disclosure, unless otherwise expressly stated, the terms "first," "second," "third," etc., are used as a nomenclature to refer only to related elements, such as equipment, components, compositions, steps, etc., and do not imply any spatial or temporal order. For example, "first equipment" and "second equipment" can refer to two separately formed devices, or two parts, components, or operating states of the same device, and can be named arbitrarily.

[0070] The terms "module," "submodule," "circuit," "subcircuit," "circuit device," "subcircuit device," "unit," or "subunit" can include memory (shared, dedicated, or grouped) storing code or instructions executable by one or more processors. A module can include one or more circuits, with or without stored code or instructions. A module or circuit can include one or more components that are directly or indirectly connected. These components may or may not be physically attached to each other or positioned adjacent to each other.

[0071] As used herein, depending on the context, the terms "if" or "when" can be understood to mean "at the time of" or "in response to". If these terms appear in the claims, they may not indicate that the relevant limitation or feature is conditional or optional. For example, a method may include the steps of: i) performing a function or action X' when or if condition X exists; and ii) performing a function or action Y' when or if condition Y exists. The method can be implemented to have both the ability to perform a function or action X' and the ability to perform a function or action Y'. Therefore, functions X' and Y' can both be performed at different times in multiple executions of the method.

[0072] A unit or module can be implemented purely in software, purely in hardware, or a combination of hardware and software. In a purely software implementation, for example, a unit or module may include functionally related code blocks or software components that are directly or indirectly linked together to perform a specific function.

[0073] Figure 1A is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. As shown in Figure 1A, system 10 includes a source device 12 that generates and encodes video data for later decoding by a target device 14. The source device 12 and the target device 14 can include any electronic device from a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some embodiments, the source device 12 and the target device 14 are equipped with wireless communication capabilities.

[0074] In some implementations, target device 14 may receive encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving encoded video data from source device 12 to target device 14. In one example, link 16 may include a communication medium enabling source device 12 to transmit encoded video data directly to target device 14 in real time. The encoded video data may be modulated according to communication standards, such as wireless communication protocols, and transmitted to target device 14. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that may facilitate communication from source device 12 to target device 14.

[0075] In some other embodiments, encoded video data may be sent from output interface 22 to storage device 32. The encoded video data in storage device 32 may then be accessed by target device 14 via input interface 28. Storage device 32 may include any data storage medium of various distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, digital versatile disks (DVDs), compact disc read-only memory (CD-ROMs), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, storage device 32 may correspond to a file server or another intermediate storage device that can hold the encoded video data generated by source device 12. Target device 14 may access the stored video data from storage device 32 via streaming or downloading. The file server may be any type of computer capable of storing and sending encoded video data to target device 14. Exemplary file servers include web servers (e.g., for websites), file transfer protocol (FTP) servers, network attached storage (NAS) devices, or local disk drives. The target device 14 can access the encoded video data via any standard data connection suitable for accessing encoded video data stored on the file server, including wireless channels (e.g., Wi-Fi connections), wired connections (e.g., Digital Subscriber Line (DSL), cable modems, etc.), or combinations thereof. The transmission of encoded video data from storage device 32 can be streaming, downloading, or a combination of both.

[0076] As shown in Figure 1A, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. The video source 18 may include sources or combinations of such sources, such as: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video. As an example, if the video source 18 is a camera in a security monitoring system, source device 12 and target device 14 may form a camera phone or video phone. However, the embodiments described in this application are generally applicable to video encoding and decoding and can be applied to wireless and / or wired applications.

[0077] Video captured, pre-captured, or computer-generated video can be encoded by video encoder 20. Encoded video data can be directly transmitted to target device 14 via output interface 22 of source device 12. Encoded video data can also (or optionally) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback. Output interface 22 may also include a modem and / or transmitter. Encoded video data may include a series of images, each image comprising one or more sample arrays, for example, for monochrome, only luminance (Y); luminance and two chrominances in the YCbCr or YCgCo domain; or green, blue, and red in the GBR (also known as RGB) domain. For ease of reference and terminology in this application, in some embodiments, variables and terms associated with each set having three sample arrays may be referred to as luminance and chrominance, where the two chrominance arrays may be referred to as Cb and Cr, regardless of the actual color representation used. Video data can be in chroma format 4:0:0, chroma format 4:2:0, chroma format 4:2:2 or chroma format 4:4:4, but this application is not limited to these.

[0078] The target device 14 includes an input interface 28, a video decoder 30, and a display device 34. The input interface 28 may include a receiver and / or a modem, and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by the video encoder 20 for use by the video decoder 30 when decoding the video data. Such syntax elements may be included within the encoded video data transmitted on a communication medium, stored on a storage medium, or stored on a file server.

[0079] In some embodiments, the target device 14 may include a display device 34, which may be an integrated display device or an external display device configured to communicate with the target device 14. The display device 34 displays decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0080] The video encoder 20 and video decoder 30 may operate according to proprietary or industry standards (e.g., VVC, HEVC, MPEG-4, Part 10, AVC) or extensions of such standards. It should be understood that this application is not limited to any particular video encoding / decoding standard and is applicable to other video encoding / decoding standards. It is generally understood that the video encoder 20 of the source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is also generally understood that the video decoder 30 of the target device 14 can be configured to decode video data according to any of these current or future standards.

[0081] The video encoder 20 and video decoder 30 can be implemented as any circuit of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-volatile computer-readable medium and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in this disclosure. Each of the video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder / decoder (CODEC) in the respective device.

[0082] In some implementations, at least some components of the source device 12 (e.g., video source 18, video encoder 20 or components included in the video encoder 20 as described below with reference to FIG. 2, and output interface 22) and / or at least some components of the target device 14 (e.g., input interface 28, video decoder 30 or components included in the video decoder 30 as described below with reference to FIG. 3, and display device 34) may operate in a cloud computing service network that provides software, platforms, and / or infrastructure, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). In some implementations, one or more components of the source device 12 and / or the target device 14 that are not included in the cloud computing service network may be provided in one or more client devices, and these one or more client devices may communicate with a server computer in the cloud computing service network via a wireless communication network (e.g., a cellular communication network, a short-range wireless communication network, or a Global Navigation Satellite System (GNSS) communication network) or a wired communication network (e.g., a local area network (LAN) communication network or a power line communication (PLC) network). In embodiments, at least a portion of the operations described herein may be implemented as a cloud-based service provided by one or more server computers in a cloud computing service network, said one or more server computers being implemented by at least some components of source device 12 and / or at least some components of target device 14; one or more other operations described herein may be implemented by one or more client devices. In some embodiments, the cloud computing service network may be a private cloud, a public cloud, or a hybrid cloud. Terms such as “cloud,” “cloud computing,” and “cloud-based” used herein may be used aptly and interchangeably without departing from the scope of this disclosure. It should be understood that this disclosure is not limited to implementation in the aforementioned cloud computing service network. Alternatively, this disclosure may also be implemented in any other type of computing environment currently known or developed in the future.

[0083] Similar to HEVC, VVC is built upon a block-based hybrid video coding and decoding framework. Figure 1B is a block diagram illustrating a block-based video encoder according to some embodiments of this disclosure. In encoder 100, the input video signal is processed block by block, called a coding unit (CU). Encoder 100 can be the video encoder 20 shown in Figure 1A. In VTM-1.0, the CU can reach 128×128 pixels. However, unlike HEVC, which is based solely on quadtree-based block segmentation, in VVC, a coding tree unit (CTU) is split into CUs to accommodate different local characteristics based on quadtree / binary / tritree. Furthermore, the concept of multiple segmentation unit types in HEVC is removed; that is, there is no distinction between CUs, prediction units (PUs), and transform units (TUs) in VVC. Instead, each CU is always used as a basic unit for both prediction and transformation without further segmentation. In a multi-type tree structure, a CTU is first segmented using a quadtree structure. Then, each quadtree leaf node can be further segmented using binary and ternary tree structures.

[0084] Figures 3A to 3E are schematic diagrams illustrating multiple tree segmentation patterns according to some embodiments of the present disclosure. Figures 3A to 3E show five segmentation types, including quadrilateral segmentation (Figure 3A), vertical binary segmentation (Figure 3B), horizontal binary segmentation (Figure 3C), vertical ternary segmentation (Figure 3D), and horizontal ternary segmentation (Figure 3E).

[0085] For each given video block, spatial prediction and / or temporal prediction can be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples (called reference samples) of already encoded neighboring blocks in the same video picture / strip to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion-compensated prediction") uses reconstructed pixels from already encoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is typically sent as a signal using one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. Additionally, if multiple reference pictures are supported, a reference picture index is sent to identify which reference picture in the reference picture storage the temporal prediction signal originates from.

[0086] Following spatial and / or temporal prediction, the intra / inter-frame mode decision circuit 121 in encoder 100 selects the optimal prediction mode, for example, based on a rate-distortion optimization method. Then, block prediction 120 is subtracted from the current video block; and the resulting prediction residual is decorrelated using transform circuit 102 and quantization circuit 104. The resulting quantization residual coefficients are dequantized by inverse quantization circuit 116 and inverse transformed by inverse transform circuit 118 to form a reconstruction residual, which is then added back to the prediction block to form the reconstructed signal of the CU. Furthermore, loop filtering 115 (such as a deblocking filter, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF)) can be applied to the reconstructed CU before it is placed into the reference picture memory of picture buffer 117 and used for encoding and decoding future video blocks. To form the output video bitstream 114, the encoding / decoding mode (inter-frame or intra-frame), prediction mode information, motion information, and quantization residual coefficients are all sent to entropy coding unit 106 for further compression and packing to form the bitstream.

[0087] For example, deblocking filters are available in the current versions of AVC, HEVC, and VVC. In HEVC, an additional loop filter (called SAO) is defined to further improve encoding and decoding efficiency. In the current version of the VVC standard, another loop filter (called ALF) is under active investigation and is very likely to be included in the final standard.

[0088] These loop filter operations are optional. Performing these operations helps improve encoding / decoding efficiency and visual quality. As a decision made by encoder 100, they can also be turned off to save computational complexity.

[0089] It should be noted that if these filter options are enabled by encoder 100, intra-frame prediction is typically based on unfiltered reconstructed pixels, while inter-frame prediction is based on filtered reconstructed pixels.

[0090] Figure 2A is a block diagram illustrating a block-based video decoder 200 that can be used in conjunction with many video codec standards. This decoder 200 is similar to the reconstruction-related portion of the encoder 100 in Figure 1B. The block-based video decoder 200 can be the video decoder 30 shown in Figure 1A. In the decoder 200, the input video bitstream 201 is first decoded via entropy decoding 202 to derive quantization coefficient levels and prediction-related information. The quantization coefficient levels are then processed via inverse quantization 204 and inverse transform 206 to obtain the reconstructed prediction residuals. The block prediction mechanism implemented in the intra / inter-frame mode selector 212 is configured to perform intra-frame prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by summing the reconstructed prediction residuals from the inverse transform 206 and the prediction output generated by the block prediction mechanism using adder 214.

[0091] Before the reconstructed blocks are stored in the image buffer 213, which serves as a reference image memory, they can also pass through the loop filter 209. The reconstructed video in the image buffer 213 can be sent to drive the display device and used to predict future video blocks. With the loop filter 209 enabled, filtering operations are performed on these reconstructed pixels to produce the final reconstructed video output 222.

[0092] Figure 1G is a block diagram illustrating another exemplary video encoder 20 according to some embodiments described in this application. The video encoder 20 can perform intra-frame predictive coding and inter-frame predictive coding of video blocks within a video frame. Intra-frame predictive coding relies on spatial prediction to reduce or remove spatial redundancy in the video data within a given video frame or picture. Inter-frame predictive coding relies on temporal prediction to reduce or remove temporal redundancy in the video data within adjacent video frames or pictures of a video sequence. It should be noted that the term "frame" can be used as a synonym for the terms "image" or "picture" in the field of video encoding and decoding.

[0093] As shown in Figure 1G, the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copying (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63, such as a deblocking filter, can be located between the adder 62 and the DPB 64 to filter block boundaries to remove block artifacts from the reconstructed video. In addition to the deblocking filter, another loop filter, such as a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF), can be used to filter the output of the adder 62. It should be noted that, regarding the CCSAO technology, this application is not limited to the embodiments described herein, but can also be applied to selecting an offset for any other component of the luminance component and the two chrominance components based on any one of the luminance component and the two chrominance components (as mentioned above, for ease of notation and terminology in this application, in the YCbCr domain, the luminance component and the two chrominance components can represent Y, Cb, and Cr; in the YCgCo domain, the luminance component and the two chrominance components can represent Y, Cg, and Co; or in the RGB domain, the luminance component and the two chrominance components can represent G, B, and R), to modify that other component based on the selected offset. Furthermore, it should be noted that the first component mentioned herein can be any one of the luminance component and the two chrominance components, the second component mentioned herein can be any other one of the luminance component and the two chrominance components, and the third component mentioned herein can be the remaining component of the luminance component and the two chrominance components. In some examples, the loop filter can be omitted, and the decoded video block can be directly provided to the DPB 64 by the adder 62. The video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the fixed or programmable hardware units described.

[0094] Video data memory 40 can store video data to be encoded by components of video encoder 20. The video data in video data memory 40 can be obtained, for example, from video source 18 as shown in FIG. 1A. DPB 64 is a buffer that stores reference video data (e.g., reference frames or pictures) for use by video encoder 20 (e.g., in intra-frame predictive coding mode or inter-frame predictive coding mode) when encoding video data. Video data memory 40 and DPB 64 can be formed from any of a variety of memory devices. In various examples, video data memory 40 can be on-chip along with other components of video encoder 20, or off-chip relative to those components.

[0095] As shown in Figure 1G, after receiving video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a collection of video blocks), or other larger coding units (CUs) according to a predefined splitting structure associated with the video data (e.g., a quadtree (QT) structure). A video frame is, or can be considered, a two-dimensional array or matrix of sample points with sample values. Sample points in the array may also be referred to as pixels or image elements (pel). The number of sample points in the horizontal and vertical directions (or axes) of the array or image defines the size and / or resolution of the video frame. For example, a video frame can be divided into multiple video blocks using QT segmentation. A video block is again, or can be considered, a two-dimensional array or matrix of sample points with sample values, but its dimension is smaller than that of the video frame. The number of sample points in the horizontal and vertical directions (or axes) of the video block defines the size of the video block. Video blocks can be further divided into one or more block partitions or sub-blocks (which can then re-form blocks) by iteratively using, for example, QT partitioning, binary tree (BT) partitioning, or ternary tree (TT) partitioning, or any combination thereof. It should be noted that the term "block" or "video block" as used herein can be a portion of a frame or image, particularly a rectangular (square or non-square) portion. Referring, for example, to HEVC and VVC, a block or video block can be or corresponds to a coding tree unit (CTU), CU, prediction unit (PU), or transform unit (TU) and / or can be or corresponds to a corresponding block (e.g., coding tree block (CTB), coding block (CB), prediction block (PB), or transform block (TB)) and / or sub-block.

[0096] The prediction processing unit 41 can select one of several feasible predictive coding modes for the current video block based on error results (e.g., coding rate and distortion level), such as one or more inter-frame predictive coding modes among multiple intra-frame predictive coding modes. The prediction processing unit 41 can provide the resulting intra-frame or inter-frame predictive coding block to adder 50 to generate a residual block, and to adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information) to entropy coding unit 56.

[0097] To select a suitable intra-predictive coding mode for the current video block, the intra-predictive processing unit 46 within the prediction processing unit 41 can perform intra-predictive coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. The motion estimation unit 42 and motion compensation unit 44 within the prediction processing unit 41 perform inter-predictive coding of the current video block relative to one or more prediction blocks in one or more reference frames to provide temporal prediction. The video encoder 20 can perform multiple coding passes, for example, to select a suitable coding mode for each block of video data.

[0098] In some implementations, motion estimation unit 42 determines an inter-frame prediction mode for the current video frame by generating motion vectors based on a predetermined pattern within the video frame sequence. The motion vectors indicate the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate the motion of a video block. For example, the motion vectors may indicate the displacement of a video block within the current video frame or picture relative to a prediction block in a reference frame relative to the current block being encoded in the current frame. The predetermined pattern may designate the video frames in the sequence as P-frames or B-frames. Intra-frame BC unit 48 may determine vectors (e.g., block vectors) for intra-frame BC coding in a similar manner to how motion estimation unit 42 determines motion vectors for inter-frame prediction, or the block vectors may be determined using motion estimation unit 42.

[0099] Regarding pixel differences, the predicted block of the video block can be, or can correspond to, a block or reference block of a reference frame that closely matches the video block to be encoded. Pixel differences can be determined by the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. In some embodiments, the video encoder 20 can compute values ​​for sub-integer pixel positions of the reference frame stored in the DPB 64. For example, the video encoder 20 can interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Therefore, the motion estimation unit 42 can perform motion search relative to full-pixel positions and fractional pixel positions and output a motion vector with fractional pixel accuracy.

[0100] The motion estimation unit 42 calculates the motion vector for a video block in an inter-frame predictive coding frame by comparing the position of the video block with the position of the predicted block in a reference frame selected from a first reference frame list (list 0) or a second reference frame list (list 1), where each of the first and second reference frame lists has an identifier stored in one or more reference frames in a DPB64. The motion estimation unit 42 sends the calculated motion vector to the motion compensation unit 44, and then to the entropy coding unit 56.

[0101] Motion compensation performed by motion compensation unit 44 may involve extracting or generating prediction blocks based on motion vectors determined by motion estimation unit 42. Upon receiving motion vectors for the current video block, motion compensation unit 44 may locate the prediction block pointed to by the motion vector in a reference frame list of reference frames, retrieve the prediction block from DPB 64, and forward the prediction block to adder 50. Adder 50 then forms a residual video block of pixel difference by subtracting the pixel values ​​of the prediction block provided by motion compensation unit 44 from the pixel values ​​of the current video block being encoded. The pixel difference forming the residual video block may include a luminance component difference or a chrominance component difference or both. Motion compensation unit 44 may also generate syntax elements associated with video blocks of a video frame for use by video decoder 30 when decoding video blocks of a video frame. Syntax elements may include, for example, syntax elements defining motion vectors for identifying prediction blocks, any flags indicating prediction modes, or any other syntax information described herein. Note that motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes.

[0102] In some implementations, the intra-BC unit 48 may generate vectors and extract prediction blocks in a manner similar to that described above in conjunction with the motion estimation unit 42 and the motion compensation unit 44; however, these prediction blocks are in the same frame as the current block being encoded, and these vectors are referred to as block vectors rather than motion vectors. Specifically, the intra-BC unit 48 may determine the intra-prediction mode to be used for encoding the current block. In some examples, the intra-BC unit 48 may, for example, use various intra-prediction modes to encode the current block during different coding passes and test their performance through rate-distortion analysis. Next, the intra-BC unit 48 may select a suitable intra-prediction mode from the various tested intra-prediction modes for use and generate an intra-mode indicator accordingly. For example, the intra-BC unit 48 may use rate-distortion analysis to calculate rate-distortion values ​​for the various tested intra-prediction modes and select the intra-prediction mode with the best rate-distortion characteristics from the tested modes as the suitable intra-prediction mode to use. Rate-distortion analysis generally determines the amount of distortion (or error) between the coded block and the original uncoded block that was encoded to generate the coded block, as well as the bit rate (i.e., the number of bits) used to generate the coded block. Intra-frame BC unit 48 can calculate the ratio from the distortion and rate for various coded blocks to determine which intra-frame prediction mode exhibits the optimal rate-distortion value for the block.

[0103] In other examples, the intra-frame BC unit 48 may use, in whole or in part, the motion estimation unit 42 and the motion compensation unit 44 to perform such functions for intra-frame BC prediction according to the embodiments described herein. In any case, for intra-frame block copying, in terms of pixel differences, the predicted block may be a block considered to closely match the block to be encoded, the pixel differences may be determined by SAD, SSD, or other difference metrics, and the identification of the predicted block may include calculating values ​​for sub-integer pixel positions.

[0104] Regardless of whether the predicted block comes from the same frame predicted intra-frame or from different frames predicted inter-frame, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the predicted block from the pixel values ​​of the current video block being encoded. The pixel difference forming the residual video block can include both luma component difference and chroma component difference.

[0105] As an alternative to the inter-frame prediction performed by the motion estimation unit 42 and the motion compensation unit 44 as described above, or the intra-block copy prediction performed by the intra-BC unit 48, the intra-prediction processing unit 46 can perform intra-frame prediction on the current video block. Specifically, the intra-prediction processing unit 46 can determine an intra-prediction mode for encoding the current block. To this end, the intra-prediction processing unit 46 can, for example, use various intra-prediction modes to encode the current block during different encoding channels, and the intra-prediction processing unit 46 (or, in some examples, a mode selection unit) can select a suitable intra-prediction mode from the tested intra-prediction modes for use. The intra-prediction processing unit 46 can provide information indicating the intra-prediction mode selected for the block to the entropy coding unit 56. The entropy coding unit 56 can encode the information indicating the selected intra-prediction mode into the bitstream.

[0106] After prediction processing unit 41 determines the prediction block for the current video block via inter-frame prediction or intra-frame prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 uses a transform (such as Discrete Cosine Transform (DCT) or a conceptually similar transform) to transform the residual video data into residual transform coefficients.

[0107] The transform processing unit 52 can send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can also reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some examples, the quantization unit 54 can then perform a scan on the matrix including the quantized transform coefficients. Optionally, the entropy coding unit 56 can perform the scan.

[0108] After quantization, the entropy coding unit 56 entropy-encodes the quantization transform coefficients into a video bitstream using, for example, Context-Adaptive Variable Length Coding (CAVLC), Context-Adaptive Binary Arithmetic Coding (CABAC), Syntax-based Context-Adaptive Binary Arithmetic Coding (SBAC), Probability Interval Partitioning Entropy (PIPE) coding, or other entropy coding methods or techniques. The encoded bitstream can then be sent to the video decoder 30 as shown in FIG. 1A, or archived in the storage device 32 as shown in FIG. 1A for later transmission to or retrieval by the video decoder 30. The entropy coding unit 56 can also entropy-encode the motion vectors and other syntax elements used for the current video frame being encoded.

[0109] The inverse quantization unit 58 and the inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct residual video blocks in the pixel domain for generating reference blocks to predict other video blocks. As noted above, the motion compensation unit 44 can generate motion-compensated prediction blocks from one or more reference blocks of a frame stored in the DPB 64. The motion compensation unit 44 can also apply one or more interpolation filters to the prediction blocks to compute sub-integer pixel values ​​for use in motion estimation.

[0110] Adder 62 adds the reconstructed residual block to the motion-compensated prediction block generated by motion compensation unit 44 to generate a reference block to be stored in DPB 64. The reference block can then be used as a prediction block by intra-frame BC unit 48, motion estimation unit 42 and motion compensation unit 44 to perform inter-frame prediction of another video block in subsequent video frames.

[0111] Figure 2B is a block diagram illustrating another exemplary video decoder 30 according to some embodiments of this application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 can perform a decoding process that is substantially the inverse of the encoding process described above in conjunction with Figure 1G regarding the video encoder 20. For example, the motion compensation unit 82 can generate prediction data based on motion vectors received from the entropy decoding unit 80, while the intra-frame prediction unit 84 can generate prediction data based on intra-frame prediction mode indicators received from the entropy decoding unit 80.

[0112] In some examples, units of the video decoder 30 may be assigned tasks to perform embodiments of this application. Furthermore, in some examples, embodiments of this disclosure may be distributed across one or more units of the video decoder 30. For example, the intra-frame BC unit 85 may perform embodiments of this application individually or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra-frame prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra-frame BC unit 85, and the functionality of the intra-frame BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0113] Video data memory 79 may store video data, such as encoded video bitstreams, that will be decoded by other components of video decoder 30. The video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source (such as a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). Video data memory 79 may include a Coded Picture Buffer (CPB) that stores encoded video data from the encoded video bitstream. The DPB 92 of video decoder 30 stores reference video data for use by video decoder 30 (e.g., in intra-frame or inter-frame prediction decoding modes) when decoding video data. Video data memory 79 and DPB 92 may be formed of any of a variety of memory devices, such as Dynamic Random Access Memory (DRAM) (including Synchronous DRAM (SDRAM)), Magnetoresistive RAM (MRAM), Resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, video data memory 79 and DPB 92 are depicted in Figure 2B as two distinct components of video decoder 30. However, it will be apparent to those skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip along with other components of video decoder 30, or off-chip relative to those components.

[0114] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of encoded video frames and associated syntax elements. The video decoder 30 may receive syntax elements at the video frame level and / or the video block level. The entropy decoding unit 80 of the video decoder 30 performs entropy decoding on the bitstream to generate quantization coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-prediction mode indicators and other syntax elements to the prediction processing unit 81.

[0115] When a video frame is encoded as an intra-predictive coded (I) frame or used as an intra-coded prediction block in other types of frames, the intra-prediction unit 84 of the prediction processing unit 81 can generate prediction data for the video block of the current video frame based on the intra-prediction mode transmitted by the signal and reference data from the previous decoded block of the current frame.

[0116] When a video frame is encoded as an inter-frame predictive coded (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the current video frame based on motion vectors and other syntax elements received from the entropy decoding unit 80. Each prediction block can be generated from a reference frame within a reference frame list. The video decoder 30 can construct the reference frame list, list 0 and list 1, based on the reference frames stored in the DPB 92 using a default construction technique.

[0117] In some examples, when decoding a video block according to the intra-BC mode described herein, the intra-BC unit 85 of the prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from the entropy decoding unit 80. The prediction block can be located within the reconstructed region of the same image as the current video block, as defined by the video encoder 20.

[0118] Motion compensation unit 82 and / or intra-frame prediction (BC) unit 85 determine prediction information for video blocks in the current video frame by parsing motion vectors and other syntax elements, and then use this prediction information to generate prediction blocks for the current video block being decoded. For example, motion compensation unit 82 uses some syntax elements from the received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) for decoding video blocks in the video frame, the inter-frame prediction frame type (e.g., B or P), construction information for one or more reference frames in the reference frame list for the frame, motion vectors for each inter-frame prediction encoded video block in the frame, inter-frame prediction state for each inter-frame prediction encoded video block in the frame, and other information for decoding video blocks in the current video frame.

[0119] Similarly, the intra-BC unit 85 may use some of the syntax elements received, such as a flag that determines whether the current video block is predicted using the intra-BC mode, construction information of which video blocks of the frame are in the reconstruction region and should be stored in the DPB92, block vectors for each intra-BC predicted video block of the frame, intra-BC prediction state for each intra-BC predicted video block of the frame, and other information for decoding video blocks in the current video frame.

[0120] The motion compensation unit 82 may also perform interpolation using interpolation filters, such as those used by the video encoder 20 during the encoding of video blocks, to calculate interpolations for sub-integer pixels of the reference block. In this case, the motion compensation unit 82 may determine the interpolation filters used by the video encoder 20 from the received syntax elements and use these interpolation filters to generate the prediction block.

[0121] The dequantization unit 86 dequantizes the quantized transform coefficients provided in the bitstream and entropy-decoded by the entropy decoding unit 80 using the same quantization parameters calculated by the video encoder 20 for each video block in the video frame to determine the degree of quantization. The inverse transform processing unit 88 applies an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to reconstruct the residual block in the pixel domain.

[0122] After the motion compensation unit 82 or the intra-frame BC unit 85 generates a prediction block for the current video block based on vectors and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra-frame BC unit 85. A loop filter 91 (such as a deblocking filter, SAO filter, CCSAO filter, and / or ALF) may be located between the adder 90 and the DPB 92 for further processing of the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be directly provided to the DPB 92 by the adder 90. The decoded video block in a given frame is then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a separate memory device may also store the decoded video for later presentation on a display device (e.g., display device 34 of FIG. 1A).

[0123] In current VVC and AVS3 standards, motion information for the current coded block is copied from spatially or temporally neighboring blocks specified by the merge candidate index, or obtained through explicit signaling of motion estimation. This disclosure focuses on improving the accuracy of motion vectors in affine merging patterns by refining the derivation method of affine merge candidates. For the sake of clarity, existing affine merging pattern designs in the VVC standard are used as examples to illustrate the proposed ideas. Note that although existing affine pattern designs in the VVC standard are used as examples throughout this disclosure, the proposed techniques can be applied to different designs of affine motion prediction patterns or other codec tools with the same or similar design principles, and will be readily apparent to those skilled in the art of modern video coding and decoding.

[0124] In a typical video encoding and decoding process, a video sequence usually consists of an ordered set of frames or images. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of chrominance samples (Cb). SCr is a two-dimensional array of chrominance samples (Cr). In other cases, a frame may be monochrome and therefore consist of only a two-dimensional array of luma samples.

[0125] As shown in Figure 1C, the video encoder 20 (or more specifically, the segmentation unit in the prediction processing unit of the video encoder 20) generates a coded representation of a frame by first dividing the frame into a set of CTUs. A video frame may include an integer number of CTUs ordered sequentially from left to right and from top to bottom in raster scan order. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by the video encoder 20 in a sequence parameter set such that all CTUs in the video sequence have the same size, which is one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that this application is not necessarily limited to a specific size. As shown in Figure 1D, each CTU may include a CTB for luma samples, two corresponding coding tree blocks for chroma samples, and syntax elements for encoding and decoding the samples of the coding tree blocks. The syntax elements describe the nature of different types of units of the coded pixel blocks and how the video sequence can be reconstructed at the video decoder 30, including inter-frame prediction or intra-frame prediction, intra-frame prediction mode, motion vectors, and other parameters. In a monochrome image or an image with three separate color planes, a CTU may include a single coding tree block and syntax elements for encoding and decoding samples of that coding tree block. The coding tree block may be an N×N sample block.

[0126] To achieve better performance, the video encoder 20 can recursively perform tree partitioning on the coding tree blocks of the CTU, such as binary tree partitioning, ternary tree partitioning, quadtree partitioning, or combinations thereof, and divide the CTU into smaller CUs. As depicted in Figure 1E, the 64×64 CTU 400 is first partitioned into four smaller CUs, each with a block size of 32×32. Among the four smaller CUs, CU 410 and CU 420 are partitioned into four CUs with a block size of 16×16, respectively. The two 16×16 CUs, CU 430 and CU 440, are further partitioned into four CUs with a block size of 8×8, respectively. Figure 1F depicts a quadtree data structure showing the final result of the partitioning process of the CTU 400 depicted in Figure 1E, where each leaf node of the quadtree corresponds to a CU of various sizes ranging from 32×32 to 8×8. Similar to the CTU depicted in Figure 1D, each CU may include two corresponding coded blocks of luma samples and chroma samples of the same size frame, and syntax elements for encoding and decoding the samples of the coded blocks. In monochrome images or images with three separate color planes, a CU may include a single coded block and syntax structures for encoding and decoding the samples of the coded block. It should be noted that the quadtree partitioning depicted in Figures 1E-1F is for illustrative purposes only, and a CTU may be split into CUs based on quadtree / tritree / binary tree partitioning to suit different local characteristics. In multi-type tree structures, a CTU is partitioned by a quadtree structure, and each quadtree leaf CU may be further partitioned by binary tree and tritree structures. As shown in Figures 3A-3E, a coded block with width W and height H has five possible partitioning types: quadruple partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal triple partitioning, and vertical triple partitioning.

[0127] In some implementations, the video encoder 20 may further divide the coded blocks of the CU into one or more M×NPBs. A PB is a rectangular (square or non-square) sample block to which the same prediction (inter-frame or intra-frame) is applied. A PU of the CU may include a PB of luma samples, two corresponding PBs of chroma samples, and syntax elements for predicting the PBs. In a monochrome image or an image with three separate color planes, a PU may include a single PB and a syntax structure for predicting the PBs. The video encoder 20 may generate predicted luma blocks, predicted Cb blocks, and predicted Cr blocks for each PU of the CU, representing the luma PB, Cb PB, and Cr PB.

[0128] Video encoder 20 can use intra-frame prediction or inter-frame prediction to generate prediction blocks for the PU. If video encoder 20 uses intra-frame prediction to generate prediction blocks for the PU, then video encoder 20 can generate prediction blocks for the PU based on decoded samples of the frame associated with the PU. If video encoder 20 uses inter-frame prediction to generate prediction blocks for the PU, then video encoder 20 can generate prediction blocks for the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0129] After the video encoder 20 generates a predicted luminance block, a predicted Cb block, and a predicted Cr block for one or more PUs of the CU, the video encoder 20 can generate a luminance residual block for the CU by subtracting the predicted luminance block of the CU from the original luminance coding block of the CU, such that each sample in the luminance residual block of the CU indicates the difference between a luminance sample in one of the predicted luminance blocks of the CU and a corresponding sample in the original luminance coding block of the CU. Similarly, the video encoder 20 can generate Cb residual blocks and Cr residual blocks for the CU, respectively, such that each sample in the Cb residual block of the CU indicates the difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and each sample in the Cr residual block of the CU indicates the difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.

[0130] Furthermore, as shown in Figure 1E, the video encoder 20 can use quadtree partitioning to decompose the luminance residual block, Cb residual block, and Cr residual block of the CU into one or more luminance transform blocks, Cb transform blocks, and Cr transform blocks, respectively. A transform block is a rectangular (square or non-square) sample block to which the same transform is applied. A TU of the CU may include a transform block for luminance samples, two corresponding transform blocks for chrominance samples, and syntax elements for transforming the samples in the transform block. Therefore, each TU of the CU may be associated with a luminance transform block, a Cb transform block, and a Cr transform block. In some examples, the luminance transform block associated with a TU may be a sub-block of the CU's luminance residual block. A Cb transform block may be a sub-block of the CU's Cb residual block. A Cr transform block may be a sub-block of the CU's Cr residual block. In a monochrome image or an image with three separate color planes, a TU may include a single transform block and syntax structures for transforming the samples in that transform block.

[0131] The video encoder 20 can apply one or more transforms to the luminance transform block of the TU to generate a luminance coefficient block for the TU. The coefficient block can be a two-dimensional array of transform coefficients. The transform coefficients can be scalars. The video encoder 20 can apply one or more transforms to the Cb transform block of the TU to generate a Cb coefficient block for the TU. The video encoder 20 can apply one or more transforms to the Cr transform block of the TU to generate a Cr coefficient block for the TU.

[0132] After generating coefficient blocks (e.g., luminance coefficient blocks, Cb coefficient blocks, or Cr coefficient blocks), video encoder 20 may quantize the coefficient blocks. Quantization typically refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After quantizing the coefficient blocks, video encoder 20 may entropy encode the syntax elements indicating the quantized transform coefficients. For example, video encoder 20 may perform CABAC on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 may output a bitstream comprising a bit sequence that forms a representation of coded frames and associated data; the bitstream may be stored in storage device 32 or transmitted to target device 14.

[0133] After receiving the bitstream generated by the video encoder 20, the video decoder 30 can parse the bitstream to obtain syntax elements. The video decoder 30 can reconstruct frames of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing the video data is generally the inverse of the encoding process performed by the video encoder 20. For example, the video decoder 30 can perform an inverse transform on the coefficient block associated with the TU of the current CU to reconstruct the residual block associated with the TU of the current CU. The video decoder 30 also reconstructs the coded blocks of the current CU by adding samples of the prediction block of the PU for the current CU to the corresponding samples of the transform block of the TU of the current CU. After reconstructing the coded blocks for each CU of the frame, the video decoder 30 can reconstruct the frame.

[0134] As mentioned above, video encoding and decoding primarily use two modes (i.e., intra-frame prediction and inter-frame prediction) to achieve video compression. It should be noted that IBC can be considered either intra-frame prediction or a third mode. Between the two modes, inter-frame prediction contributes more to encoding and decoding efficiency than intra-frame prediction because it uses motion vectors to predict the current video block from a reference video block.

[0135] However, with continuously improving video data capture technologies and increasingly finer video tile sizes used to preserve details in video data, the amount of data required to represent the motion vector of the current frame has also increased significantly. One way to overcome this challenge is to take advantage of the fact that not only do a set of adjacent CUs in both the spatial and temporal domains have similar video data for prediction purposes, but the motion vectors between these adjacent CUs are also similar. Therefore, the motion information of spatially adjacent CUs and / or temporally co-located CUs can be used as an approximation of the motion information (e.g., motion vector) of the current CU by exploring the spatial and temporal correlations of spatially adjacent CUs and / or temporally co-located CUs. This is also known as the "Motion Vector Predictor (MVP)" of the current CU.

[0136] Instead of encoding the actual motion vector of the current CU determined by the motion estimation unit into the video bitstream as described above in conjunction with Figure 1B, the motion vector prediction factor of the current CU is subtracted from the actual motion vector of the current CU to generate the motion vector difference (MVD) of the current CU. By doing so, it is not necessary to encode the motion vector determined by the motion estimation unit for each CU of the frame into the video bitstream, and the amount of data used to represent motion information in the video bitstream can be significantly reduced.

[0137] For example, in the process of selecting a prediction block in a reference frame during inter-frame prediction of a coded block, both the video encoder 20 and the video decoder 30 need to employ a set of rules to construct a motion vector candidate list (also known as a "merging list") for the current CU using those potential candidate motion vectors associated with spatially adjacent CUs and / or temporally co-located CUs. Then, a member is selected from the motion vector candidate list as the motion vector predictor for the current CU. By doing so, it is not necessary to send the motion vector candidate list itself from the video encoder 20 to the video decoder 30, and the index of the selected motion vector predictor within the motion vector candidate list is sufficient for both the video encoder 20 and the video decoder 30 to use the same motion vector predictor within the motion vector candidate list for encoding and decoding the current CU.

[0138] Some embodiments of this disclosure further enhance inter-frame encoding / decoding efficiency by applying adaptive enhancement filters to the motion-compensated prediction signal of the bidirectional prediction block. Some embodiments of this disclosure aim to further enhance the chroma encoding / decoding efficiency of the motion compensation module applied in ECM. Below, some relevant encoding / decoding tools used in the transform and entropy coding processes in ECM are briefly reviewed. Finally, some shortcomings in existing motion compensation designs are discussed. Finally, solutions to improve existing designs are proposed.

[0139] Motion Compensated Prediction (MCP)Motion-compensated prediction (MCP), also known simply as motion compensation, is one of the most widely used video coding techniques in modern video coding standards. In MCP, a video frame is divided into multiple blocks (called prediction units (PUs)). Each PU is predicted from a block of the same size from a time reference picture, significantly reducing the overhead required to signal the block. In all existing video codec standards, each inter-frame PU is associated with a set of motion parameters consisting of one or two MVs and a reference picture index. Inter-frame PUs in P-strips have only one list of reference pictures, while PUs in B-strips can use up to two lists of reference pictures. In MCP, the corresponding inter-frame prediction samples are generated from their corresponding regions in the reference picture, as identified by the MV and the reference picture index. The MV specifies the horizontal and vertical displacement between the current block and its reference block in the reference picture. Figure 4 shows d x and d y This is an example of the horizontal and vertical values ​​of an MV. In practice, an MV value can be fractional in precision. As shown in Figure 5, when an MV has a fractional value, an interpolation filter is applied to generate the corresponding predicted sample at the fractional sample location. In VVC, for luminance MC, MV in units of 1 / 16 of the distance between two adjacent luminance samples is supported; for chrominance MC, MV in units of 1 / 32 of the distance between two adjacent chrominance samples is supported.

[0140] Adaptive Loop Filtering In VVC and ECM, one of 25 filters is selected for each 4×4 block in Adaptive Loop Filtering (ALF) based on the direction and activity of the local gradient.

[0141] Filter shape: Two diamond filter shapes are used (as shown in Figures 6A to 6B). A 7×7 diamond shape is applied to the luminance component, and a 5×5 diamond shape is applied to the chrominance component.

[0142] Block classification: For the luminance component, each 4×4 block is classified into one of 25 categories. The classification index C is based on its directionality. And the quantitative value of activity Export, as shown below:

[0143] In order to calculate and First, the gradients in the horizontal, vertical, and two diagonal directions are calculated using the 1-D Laplacian operator:

[0144] Among them, index and This represents the coordinates of the top-left sample point within a 4×4 block. Indicator at coordinates Reconstructed sample points at the location. To reduce the complexity of block classification, as shown in Figure 7, downsampling 1-D Laplacian operator computation is applied to gradient calculation in all directions.

[0145] Then, the maximum and minimum values ​​of the gradient of D in the horizontal and vertical directions are set as follows:

[0146] The maximum and minimum values ​​of the gradients in the two diagonal directions are set as follows:

[0147] To derive directionality The values ​​are compared with each other and with two thresholds. and Comparison: Step 1. If If both are true, then D is set to 0.

[0148] Step 2. If If yes, continue from step 3; otherwise, continue from step 4.

[0149] Step 3. If If the condition is met, then D is set to 2; otherwise, D is set to 1.

[0150] Step 4. If If the value is 4, then D is set to 4; otherwise, D is set to 3.

[0151] Activity value Calculated as:

[0152] Further quantization is performed to a range of 0 to 4 (inclusive), and the quantized value is expressed as... For the chromaticity components in an image, classification methods should not be applied.

[0153] Geometric transformation of filter coefficients and clipping values Before filtering each 4×4 luminance block, geometric transformations such as rotation or diagonal and vertical flipping are applied to the filter coefficients, depending on the gradient values ​​calculated for the block. and the corresponding filter clipping value This is equivalent to applying these transformations to samples in the filter's support region. The idea is to make the different blocks more similar by aligning the directions of the blocks to which the ALF is applied.

[0154] It provides three geometric transformations: diagonal, vertical flip, and rotation.

[0155] in, It is the size of the filter. , These are coefficient coordinates, which make the position... In the top left corner, the location In the bottom right corner. Depending on the gradient values ​​computed for the block, the transformation is applied to the filter coefficients f(k, l) and the clipping value. The relationship between the transformation and the four gradients in the four directions is summarized in Table 1.

[0156]

[0157] Table 1 Filtering process When ALF is enabled for CTB, each sample point within CU It will be filtered to obtain sample values. As shown below,

[0158] in, Indicates the filter coefficients used for decoding. For the clipping function, This represents the clipping parameters for decoding. Variables k and l are... and Between, where L represents the filter length. This is the clipping function, which clips the input value of x to the range [-y, y]. The clipping operation introduces non-linearity to make ALF more efficient by reducing the influence of neighboring sample values ​​that differ too much from the current sample value.

[0159] Local illumination compensation Local Illumination Compensation (LIC) is an encoding / decoding tool researched during the development of VVC, aiming to address local illumination variations present in temporally adjacent images. LIC is based on a linear model that derives scaling factors and offsets to enhance predicted samples of the current block. Specifically, LIC can be mathematically modeled as follows:

[0160] in, The current block is at coordinates Predicted signal at the location; Based on motion vectors The generated prediction block; and These are the corresponding scaling factors and offsets. Figure 8 illustrates the LIC process. As shown in Figure 8, when LIC is applied to a video block, a linear model (i.e., the scaling factor) is derived by minimizing the difference between the neighboring samples of the current block (i.e., the template in Figure 8) and their corresponding predicted samples (i.e., the template prediction in Figure 8). and offset ).

[0161] Because the scaling factor and offset are derived based on the current block and template and their corresponding prediction signals, no signaling overhead for LIC parameters is required. Additionally, a LIC flag is signaled for an unmerged inter-block to indicate whether LIC mode is enabled for that block. For merged inter-blocks, the LIC flag is considered part of the motion information. Specifically, when the merge list is built, the LIC flag is inherited from the LIC flags of its corresponding neighboring blocks, in addition to the MV and reference index being derived from the MV and reference index of their respective neighboring blocks. Simultaneously, LIC mode is also applied to affine inter-blocks. When affine mode is applied, an inter-block is divided into multiple sub-blocks, and a specific MV is derived for each sub-block based on the affine model. Based on this design, when LIC is applied to an affine block, the corresponding LIC parameters are derived based on the motion information of the sub-blocks on the top and left boundaries of the block; then, the derived LIC model is applied to the prediction samples of the entire block, as shown in Figure 9. Since the MV of each boundary sub-block may be different, the prediction signal of the template is also generated based on the sub-blocks, and the prediction samples of each template sub-block are generated using the MV of the corresponding sub-block on the coded block boundary.

[0162] Finally, it should be noted that in the current LIC design, LIC is only applicable to inter-frame blocks with unidirectional prediction.

[0163] Bidirectional prediction with CU-level weights In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference images and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow a weighted average of the two prediction signals, i.e.,

[0164] Five weights are allowed in weighted average two-way forecasting. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) for non-merging CUs, the weight index is sent using a signal; 2) for merged CUs, the weight index is inherited from one of the neighboring blocks based on the merge candidate index. Additionally, in VVC, all 5 weights are used for low-latency images (i.e., all reference images precede the current image in display order). Otherwise, for non-low-latency images (where there is at least one reference image following the current image in display order), only 3 weights (w∈ {3,4,5}) are used.

[0165] Overlapping block motion compensation OBMC is a codec technique that removes block artifacts during the MC stage. The basic idea of ​​OBMC is to perform motion compensation on the current block using the motion capture (MV) from neighboring blocks, and then combine multiple prediction signals using the neighboring MVs to generate the final prediction signal for the control unit (CU). For each inter-frame CU, OBMC is performed on the top and left boundaries of the block. Additionally, when a video block is encoded and decoded in a sub-block mode (e.g., affine, ATMVP, or DMVR), OBMC is also performed on all internal boundaries of each sub-block (i.e., top, left, bottom, and right boundaries). Figure 15 illustrates the OBMC process applied to a CU without sub-block-level motion compensation. When OBMC is applied to a sub-block (e.g., sub-block A in Figure 15), in addition to the left and top neighbors of the sub-block, the MVs of the neighboring sub-blocks on the right and bottom of the current sub-block are also used to derive the prediction signal; then, the four prediction blocks are averaged to generate the final prediction signal for the current sub-block.

[0166] The current ECM software employs a template-based OBMC scheme. Specifically, the method for deriving the predicted values ​​of CU boundary samples does not use fixed weights for a combination of multiple motion compensation hypotheses, but rather determines them based on the template matching cost, including using only the motion information of the current block, or using the motion information of neighboring blocks, or a hybrid mode.

[0167] In this scheme, for each block with a 4×4 size at the top CU boundary, the template size is equal to 4×1. If N adjacent blocks have the same motion information, the template size is enlarged to 4N×1 because the MC operation can be performed in one go. For each left block with a 4×4 size at the left CU boundary, the left template size is equal to 1×4 or 1×4N (as shown in Figure 16).

[0168] For each 4×4 top block (or N groups of 4×4 blocks), follow these steps to derive the predicted values ​​for the boundary samples.

[0169] Taking block A as the current block and its adjacent block AboveNeighbor_A as an example, the operations on the left-hand blocks are performed in the same way.

[0170] First, based on the following three types of motion information, the SAD between the template reconstruction sample points derived from the MC process and their corresponding reference sample points is used to measure the three template matching costs. , , ): It is calculated based on the motion information of A.

[0171] It is calculated based on the motion information of AboveNeighbor_A.

[0172] It is calculated based on a weighted prediction of the motion information of A and AboveNeighbor_A, where the weighting factors are 3 / 4 and 1 / 4, respectively.

[0173] Secondly, a method is selected to calculate the final prediction results of the boundary samples by comparing Cost1, Cost2, and Cost3.

[0174] The original MC result using the motion information of the current block is represented as follows: And the MC result using the motion information of neighboring blocks is represented as The final prediction result is expressed as .

[0175] If Cost1 is the minimum, then .

[0176] If (Cost2 + (Cost2>>2) + (Cost2>>3))<= Cost1, then use mixed mode 1.

[0177] For a luminance block, the number of mixed pixel rows is 4.

[0178]

[0179]

[0180]

[0181]

[0182] For chroma blocks, the number of mixed pixel rows is 1.

[0183]

[0184] If Cost1 <= Cost2, then use Mixed Mode 2.

[0185] For a luminance block, the number of mixed pixel rows is 2.

[0186]

[0187]

[0188] For chroma blocks, the number of mixed pixel rows / columns is 1.

[0189]

[0190] Otherwise, use mixed mode 3.

[0191] For a luminance block, the number of mixed pixel rows is 4.

[0192]

[0193]

[0194]

[0195] For chroma blocks, the number of mixed pixel rows is 1.

[0196]

[0197] Adaptive reordering of merged candidates using template matching In ECM, a reordering tool called Adaptive Reordering of Merge Candidates Using Template Matching (ARMC) is applied to the merging mode of inter-frame encoding and decoding. When this method is applied, the merge candidates are adaptively ordered according to the template matching (TM) cost. This method is applicable to both regular merging mode and affine merging mode.

[0198] Specifically, in the ARMC design, an initial merge candidate list is first constructed, which includes multiple merge candidates, such as spatial, TMVP, non-adjacent, HMVP, and pairwise merge candidates. The candidates in the initial list are then divided into one or more subgroups. The merge candidates in each subgroup are reordered according to a template-matching cost to generate a reordered merge candidate list. The indices of the selected merge candidates in the reordered merge candidate list are then signaled from the encoder to the decoder.

[0199] During the reordering process, the template matching cost of the merge candidates is measured by the SAD between the template sample of the current block and its corresponding reference sample. The template includes a set of reconstructed samples adjacent to the current block. The reference sample of the template is located using the motion information of the merge candidates. When the merge candidates utilize bidirectional prediction, the reference sample of the merge candidate's template is also generated through bidirectional prediction, as shown in Figure 18.

[0200] For affine patterns, since different sub-blocks can represent different motion vectors, the predicted samples of the template are generated based on sub-block-based motion compensation. Specifically, as shown in Figure 19, assuming the size of the candidate sub-blocks for sub-block-based affine merging is equal to Wsub × Hsub, the upper template includes several sub-templates of size Wsub × 1, and the left template includes several sub-templates of size 1 × Hsub. The motion information of the sub-blocks in the first row and first column of the current block is used to derive the reference samples for each sub-template.

[0201] Merging patterns based on template matching and MVD Besides the implicitly derived motion information directly used in the merging pattern for generating prediction samples in the current CU, the Motion Vector Difference (MMVD) merging pattern is also applied to both the regular merging pattern and the affine merging pattern. For signaling, an MMVD flag is sent immediately after the regular merging flag to indicate whether the MMVD pattern is used for the CU. In the ECM, 16 refined positions are defined along the k×π / 8 diagonal angle for the MMVD pattern, as shown in Figure 20. Additionally, the top N motion candidates in the candidate list before reordering are used as the base candidates for both MMVD and affine MMVD. For MMVD, N equals 3; for affine MMVD, N equals 1 or 3 depending on the affine flag of the neighboring blocks. When a base candidate is predicted bidirectionally, two ways of adding MMVD offsets are allowed: "both sides" and "one side". In the "two-sided" MMVD mode, depending on the POC relationship between the current image and its reference images in L0 and L1, the same selected MMVD offset (or its opposite) is applied to both the candidate L0 MV and L1 MV. In the "one-sided" MMVD mode, the selected MMVD offset is applied to the MV in only one reference image list (L0 or L1), while the MV in the other reference list remains unchanged. Accordingly, based on this design, there are a total of 16 × 6 × 3 = 288 refinement positions for MMVD mode. To save signaling overhead, all 288 possible refinement positions are reordered based on the SAD cost between the template (one row above and one column to the left of the current block) and its reference for each refinement position, and only the first 36 refined positions after reordering are allowed to be selected, indicated by an MMVD index in the bitstream.

[0202] AMVP Merge Mode In ECM, a novel bidirectional coding mode is introduced, consisting of an Advanced Motion Vector Prediction (AMVP) predictor in one direction and a merge predictor in the other. This mode is enabled for a coding block when the selected merge predictor and AMVP predictor meet the following conditions: one reference image is from the past, and one is from the future, and both reference images are equidistant from the current image. If bilateral matching is enabled, bilateral matching MV refinement is applied to both the merge MV candidate and the AMVP MVP, which serve as the starting point. Otherwise, if template matching is enabled, template matching MV refinement is applied to either the merge predictor or the AMVP predictor, depending on which has the higher template matching cost.

[0203] In this mode, the AMVP portion is signaled as a regular one-way AMVP, that is, the reference index and MVD are signaled, and if template matching is used, it has the derived MVP index, or when template matching is disabled, the MVP index is signaled.

[0204] For the AMVP direction LX, where X can be 0 or 1, the merging portion in the other direction (1-LX) is implicitly derived by minimizing the bilateral matching cost between the AMVP predictor and the merging predictor (i.e., for a pair of AMVPs and the merging motion MV). For each merging candidate in the merging candidate list for that other direction (1-LX) motion vector, the bilateral matching cost is computed using the merging candidate MV and the AMVP MV. The merging candidate with the minimum cost is selected. Bilateral matching refinement is applied to the encoded block starting with the selected merging candidate MV and the AMVP MV.

[0205] The new bidirectional encoding / decoding mode is indicated by a flag, and if the mode is enabled, the AMVP direction LX is further indicated by another flag.

[0206] When using the bilateral matching (BM) AMVP-merge mode for the current block and template matching is enabled, no MVD is signaled. An additional pair of AMVP-merge MVPs is introduced. The merge candidate list is sorted in ascending order based on the BM cost. A signaled index (0 or 1) indicates which merge candidate from the sorted merge candidate list to use. When only one candidate exists in the merge candidate list, the pair of AMVPs and merge MVPs without bilateral matching MV refinement are populated.

[0207] Combined intra-frame and inter-frame predictionIn VVC, when encoding and decoding a CU in merged mode, if the CU contains at least 64 luma samples (i.e., the width of the CU multiplied by the height of the CU is equal to or greater than 64) and if both the width and height of the CU are less than 128 luma samples, an additional flag is sent to indicate whether to apply Combined Intra-Inter-Frame Prediction (CIIP) mode to the current CU. In CIIP mode, the prediction signal is obtained by combining the inter-frame prediction signal with the intra-frame prediction signal.

[0208] The inter-frame prediction signal in CIIP mode is derived using the same inter-frame prediction procedure applied in the regular merging mode; and the intra-frame prediction signal in CIIP mode is derived after utilizing the regular intra-frame prediction procedure in the planar mode. Then, a weighted average is used to combine the intra-frame prediction signal and the inter-frame prediction signal, where the weight values ​​are calculated based on the coding modes of the top neighbor block and the left neighbor block of the current CU, as follows: if the top neighbor block is available and intra-coded, isIntraTop is set to 1, otherwise isIntraTop is set to 0; if the left neighbor block is available and intra-coded, isIntraLeft is set to 1, otherwise isIntraLeft is set to 0; if (isIntraLeft + isIntraTop) equals 2, the weight value is set to 3; otherwise, if (isIntraLeft + isIntraTop) equals 1, the weight value is set to 2; otherwise, the weight value is set to 1. The prediction signal is derived as follows:

[0209] Here, Pinter is the inter-prediction signal in CIIP mode, Pintra is the intra-prediction signal in CIIP mode, wt is the weight value, and >> indicates a right shift operation. Furthermore, when LIC is enabled, the generation of inter-prediction samples for CIIP mode always bypasses the LIC process; that is, scaling and offset are not applied to adjust the inter-prediction samples before mixing them with the intra-prediction samples.

[0210] Motion Compensation Prediction (LCP) plays a crucial role in ensuring inter-frame coding efficiency across all existing video codec standards. Using MCP, the video signal to be encoded / decoded is predicted from a temporally adjacent signal, and only the prediction error, MV (Motion Video Rendering), and reference frame index are transmitted. As previously analyzed, ALF (Aspect Ratio Rendering) can effectively improve the quality of the reconstructed video, thereby improving inter-frame coding / decoding performance by providing a high-quality reference picture. LIC can be considered an enhancement to conventional motion compensation prediction. Although both tools can enhance inter-frame coding / decoding efficiency, the quality of temporal prediction may still be insufficient due to the fact that the video signal can be encoded / decoded using coarse quantization (i.e., high quantization parameter (QP) values). When coarse quantization is applied, the reconstructed picture may contain severe coding / decoding artifacts, such as block artifacts, ringing artifacts, etc. Assuming the reconstructed signal of the current picture will be used as a reference for temporal prediction, this distortion may reduce the effectiveness of MCP, thereby reducing the inter-frame coding / decoding efficiency of subsequent pictures.

[0211] While LIC can effectively compensate for lighting variations between different images, it can only be applied to unidirectional prediction blocks. It is known that combining multiple prediction blocks can effectively suppress encoding / decoding noise present in motion-compensated signals (caused by the quantization / dequantization process). Therefore, bidirectional prediction is generally more compression-efficient than unidirectional prediction; that is, there are more bidirectional prediction blocks than unidirectional prediction blocks. This means that unidirectional LIC cannot fully utilize the potential encoding / decoding gains that LIC tools can achieve.

[0212] According to the existing OBMC design in ECM, OBMC is always disabled for inter-frame CUs using LIC encoding / decoding. Given the existence of block artifacts between inter-frame blocks encoded / decoded with and without LIC, this design is suboptimal in terms of encoding / decoding efficiency. Furthermore, even when LIC is applied to two adjacent blocks, potential block artifacts can exist along the block boundaries of the two blocks because the LIC parameters applied to the two blocks can be different.

[0213] This disclosure proposes a method and apparatus for improving the efficiency of motion compensation and thus enhancing the quality of time prediction. Specifically, it proposes applying adaptive filtering at the prediction samples of a bidirectional prediction block. To reduce signaling overhead, filter coefficients are derived from the neighboring reconstructed samples (i.e., templates) and their corresponding prediction samples of the current block. In this way, the energy of the prediction residuals is mitigated, thereby reducing the overhead of residual signaling.

[0214] Figure 10 shows a block diagram of the video encoder when the proposed adaptive bidirectional predictive filter is applied. First, similar to a conventional video encoder, the motion estimation and compensation module generates a motion-compensated signal by matching the current block with one block (unidirectional prediction) or two blocks (bidirectional prediction) of the reference image using the optimal MV. Then, for the bidirectional prediction block, motion-compensated samples (both luma and chroma) are provided to the proposed adaptive filter to generate filtered motion-compensated prediction samples for the current block. The original signal is then subtracted from the prediction signal to remove temporal redundancy and generate the corresponding residual signal. Transformation and quantization are applied to the residual signal, which is then entropy-coded and output to the bitstream. To obtain the reconstructed signal, the residual signal is reconstructed using inverse quantization and inverse transform. The reconstructed residual is then added to the motion-compensated prediction. Furthermore, loop filtering processes (e.g., deblocking, ALF, and SAO) are applied to the reconstructed video signal for output. As discussed later, the filter coefficients of the proposed adaptive bidirectional predictive filter are derived directly from the neighboring reconstructed luma and chroma samples at the decoder. In addition, to maximize the encoding and decoding gain of the proposed method, additional syntax can be signaled at a given block level (e.g., CTU, CU, or PU level) to indicate whether the proposed filter is applied to the current block for motion compensation.

[0215] Figure 11 shows a block diagram of the proposed decoder that receives the bitstream generated by the encoder in Figure 10. At the decoder, the bitstream is first parsed by an entropy decoder. The residual coefficients are then dequantized and inversely transformed to obtain the reconstructed residuals. For time prediction, a prediction signal is first generated by obtaining motion-compensated blocks using the prediction information transmitted by the signal (i.e., MV and reference index). Then, for bidirectional prediction blocks, they are parsed from the bitstream to determine whether adaptive filtering is enabled for that block. If adaptive filtering is enabled, the motion-compensated luminance and chrominance signals are further processed by the proposed adaptive filtering; otherwise, the motion-compensated chrominance signal is not filtered. The motion-compensated signal (filtered or unfiltered) and the reconstructed residuals are then added together to obtain the reconstructed video. The reconstructed video may also be loop-filtered before being stored in a reference image memory for display and / or for decoding future video signals.

[0216] Adaptive bidirectional predictive filtering based on template bidirectional predictive samplesThis section proposes an adaptive filtering scheme for bidirectional prediction, where filter coefficients are derived based on bidirectional prediction samples of a template for a bidirectional prediction block. Specifically, in the proposed scheme, bidirectional prediction samples of the template are first generated based on the motion vector of the current block; then, the least squares mean error (LMSE) algorithm is applied to derive the filter parameters by minimizing the difference between the template prediction samples and the template samples. Figure 12 illustrates the proposed template-based adaptive filtering method for bidirectional prediction samples. As shown in Figure 12, T indicates the template of the current bidirectional prediction block; and These are the L0 and L1 predicted samples of the template, which are obtained by using the bidirectional motion vector of the current block. and To generate. Based on these representations, in the proposed scheme, bidirectional prediction samples of the template are first generated by averaging the two unidirectional predictions of the template in L0 and L1, i.e.,

[0217] in, and These are the weights applied in the L0 and L1 directions when generating bidirectional prediction samples for the current block. Without BCW, the weights are equal to 0.5; with BCW, the weights can be -0.125, 0.375, 0.625, and 1.125. Based on the bidirectional prediction samples obtained from the template, LMSE derives values ​​for the coefficients of the adaptive filter to minimize the difference between the template samples and their bidirectional prediction samples.

[0218] in, This indicates that it is applied to a template prediction sample. Correspondence The coefficients of the filter in the neighboring region, where, , In practice, various filters of different sizes and shapes can be applied, offering different trade-offs between encoding / decoding performance and complexity. Larger filters can make the template prediction samples closer to the template samples, but at the cost of increased computational complexity. Finally, the derived filter coefficients are applied to modify the original bidirectional prediction signal of the current block to...

[0219] in, and These are bidirectional predicted samples before and after the application of the proposed adaptive filtering. Furthermore, to further improve the encoding / decoding gain, an offset and a specific nonlinear term can be introduced when deriving the filter coefficients in the proposed method, which can further reduce the distortion between the template samples and their predicted samples. Specifically, with such modification, the filter coefficients in (11) are derived as follows:

[0220] Furthermore, the filter in (12) is applied as follows:

[0221] in, It's an offset. It is a nonlinear term, which is represented as a template prediction sample. The sum of a series of powers (i.e., k = 2, …, K-1).

[0222] In one or more examples, it is proposed to use a linear model (i.e., scaling factor and offset) to derive a two-tap filter to enhance the prediction samples of a bidirectional prediction block. Specifically, a bidirectional prediction LIC is proposed, which operates as follows: 1) generating bidirectional prediction samples of a template, as shown in (10); 2) deriving the scaling factor and offset using the template samples and their corresponding bidirectional prediction samples.

[0223] in, and These are the scaling factor and offset of the LIC linear model; This refers to the number of template samples involved in the export. Afterwards, the final bidirectional prediction for the current block is generated as follows:

[0224] Adaptive bidirectional predictive filtering based on template unidirectional samples This section proposes an adaptive bidirectional prediction filtering scheme for unidirectional prediction samples using a template of a bidirectional prediction block. For example, in this method, two adaptive filter operations are applied to the template's prediction samples in a one-sided manner: two sets of filter coefficients are derived and applied to the prediction samples in L0 and L1 respectively; then, the weighted average of the two filtered unidirectional prediction samples is formed as the final prediction sample for the current block. Figure 13 illustrates the proposed scheme. As shown in Figure 13, based on L0 MV and L1 MV, two unidirectional predictions of the template are generated. and Then, based on and Between, and and Minimizing the distortion between the two directions separately allows for the derivation of two sets of filter parameters for the L0 and L1 directions, respectively. and It is described as:

[0225] Where N represents the number of template samples involved; These are the template sample points of the current block; This represents a one-way prediction of template samples based on the MV (L0 or L1) of the current block. Then, two filters are applied to the two one-way predictions of the current block respectively, and then combined to generate the final bidirectional prediction for the current block.

[0226] in

[0227] in, and These are the two unidirectional prediction samples of the current block before the application of the proposed adaptive filtering. Similar to (13) and (14), in addition, to further improve the encoding and decoding gain, offset and nonlinear terms can be introduced when deriving the filter coefficients. With such modifications, the filter coefficients are derived as follows:

[0228] Furthermore, the filtered one-way prediction samples of the current block are calculated as follows:

[0229] In one or more examples, a linear model (i.e., scaling factor and offset) is proposed to derive a two-tap filter to enhance the two unidirectional predictions of a bidirectional prediction block. Specifically, a bidirectional prediction LIC is proposed, which operates as follows: 1) generating two unidirectional predictions of a template; 2) deriving two sets of scaling factors and offsets using template samples and their corresponding unidirectional prediction samples.

[0230] in, and These are the scaling factors and offsets for the LIC linear model used for L0 unidirectional prediction, and and These are the scaling factor and offset for the LIC linear model used for L1 unidirectional prediction; This refers to the number of template samples involved in the derivation. Afterwards, the final bidirectional prediction for the current block is generated as...

[0231] in, and It is the BCW weight applied to the current block.

[0232] Recursive unidirectional filtering based on adaptive bidirectional predictive filtering In Figure 13, because the filter coefficients of the two unidirectional prediction signals applied to the template are derived separately, the resulting bidirectional prediction signal of the template may not be optimal (i.e., a weighted combination of the two filtered unidirectional prediction signals) when considering minimizing the distortion between the template sample and its corresponding prediction sample. To address this problem, an iterative scheme is proposed to derive the optimal filter coefficients of the two unidirectional prediction signals applied to the template for a bidirectional prediction block. The proposed scheme proceeds iteratively, alternately optimizing the prediction filter for one prediction direction while keeping the prediction filter for the other prediction direction fixed. Specifically, the derivation process of the two unidirectional prediction filter coefficients is summarized as follows: Step 1: Given the initial prediction direction By making it unidirectional prediction With template Minimize the distortion between them to derive the initial filter coefficients for the initial prediction direction. ,Right now,

[0233] Step 2: Based on filter coefficients Calculate the filtered one-way prediction And set k = 1.

[0234]

[0235] Step 3: Select the target prediction direction And calculate the target template sample points of the current block as

[0236] Step 4: By making it a one-way prediction With template To derive the initial prediction direction by minimizing the distortion between the two sides. filter coefficients ,Right now,

[0237] Step 5: Based on filter coefficients Filtered one-way prediction Calculated as

[0238] Step 6: Set k = k+1 and go to step 3.

[0239] The resulting filter is used as a corresponding filter for the two unidirectional predictions applied to the current block, and then the filtered prediction samples are combined to generate the final bidirectional prediction for the current block, as shown in (18) and (19). Similarly, the offset and nonlinear terms shown in (20) and (21) can also be applied to the proposed iterative bidirectional prediction filter derivation scheme. In addition, in one or more examples, it is proposed to use a linear model (i.e., scaling factors and offsets) to derive a two-tap filter through the proposed iterative filter derivation scheme: 1) generate two unidirectional predictions for the template; 2) derive two sets of scaling factors and offsets based on the iterative algorithm shown in steps 1 to 6; 3) compute the final bidirectional prediction samples for the current block as shown in (23).

[0240] In practice, different numbers of iterations can be applied to the iterative filter derivation scheme described above. Generally, more iterations result in less distortion between the template and its predicted signal (i.e., better encoding / decoding gain); however, this comes at the cost of increased computational complexity. Below, different methods are proposed to determine the number of iterations applied in the proposed algorithm. One method proposes using a fixed number of iterations (i.e., 3) at both the encoder and decoder. A second method proposes giving the encoder the freedom to choose a specific number of iterations and sending the corresponding value to the decoder as a signal. When applying this method, new syntax elements (or multiple new syntax elements) can be added at the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), picture header, strip header, or even at the code block level to indicate the value of the applied iterations. A third method proposes adaptively determining the iteration value applied to a block based on block statistics (e.g., sample changes, motion vector differences, and additional data). In one or more examples, the difference between the original L0 predicted samples and the original L1 predicted samples of a bidirectional prediction block is proposed as a criterion for selecting the number of iterations applied. For example, when the difference between two predicted samples (i.e., the sum of absolute differences (SAD), the sum of squared differences (SSD), and other matrices) is greater than a threshold, more iterations are applied to that block; otherwise (i.e., the difference is less than the threshold), fewer iterations are applied.

[0241] Last but not least, different initial prediction directions can be applied in the proposed schemes. One approach proposes always using L0 as the initial prediction direction. Another approach proposes using L1 as the initial prediction direction. A third approach proposes selecting the initial prediction direction based on the stripe type, prediction structure, and QP of the current block. For example, it could use L0 as the initial prediction direction for non-low-latency images and L1 as the initial prediction direction for low-latency images.

[0242] Signaling for adaptive motion compensation filteringIn practice, various signaling schemes can be applied to instruct the use of the proposed adaptive motion compensation filtering for bidirectional prediction inter-frame blocks. In one embodiment of this disclosure, for explicit inter-frame mode (i.e., AMVP mode), a control flag is proposed to be signaled to explicitly indicate whether adaptive motion compensation filtering is applied to the current block. When the flag is one, it indicates that adaptive filtering is applied to motion compensation prediction samples, and the corresponding filter coefficients are derived from the template samples using one of the methods discussed above. Otherwise, when the flag is zero, it indicates that adaptive filtering is not applied to the current block. On the other hand, for merging mode, it is proposed that, in addition to inheriting other motion information (e.g., MV, reference index, and additional information) from its corresponding selected merging candidate (as shown in the merging index), its control flag is also inherited from its corresponding selected merging candidate (as shown in the merging index).

[0243] Multi-template adaptive motion compensation filtering As described above, when the proposed adaptive motion compensation filter is applied, template samples above and to the left of the current block (when they are available) are used to derive the filter coefficients of the block. This may not always be optimal due to the varying correlations between the current block and its top / left neighbors. To address this issue, two additional adaptive filtering modes are proposed: in the first mode (referred to as the top-only filtering mode), only the neighboring samples above the current block (i.e., the top template) are used to derive the filter coefficients; in the second mode (referred to as the left-only filtering mode), only the neighboring samples to the left of the current block (i.e., the left template) are used to derive the filter coefficients of the current block. For signaling, a flag can be sent first to indicate whether the default filtering mode (i.e., using both top and left neighboring samples) is applied. When the flag is false (i.e., the default filtering mode is not applied), another flag can be sent to indicate which mode (top-only mode or left-only mode) is applied to the current block. In one approach, the two proposed additional multi-template filtering modes are applied to all inter-frame blocks. In another approach, the two proposed additional multi-template filtering modes are applied only to AMVP inter-frame blocks. In yet another approach, the two proposed additional multi-template filtering modes are applied only to unidirectionally predicted AMVP inter-frame blocks.

[0244] In addition, in a specific example, it is proposed to apply the proposed multi-template pattern to an existing LIC design, where the proposed adaptive motion compensation filter degenerates into a 2-tap filter (i.e., a scaling and an offset).

[0245] Adaptive motion compensation filtering based on non-adjacent spatial neighborsIn some embodiments, the blocks surrounding the current block are defined as neighboring blocks of the current block. As shown in Figures 17A and 17B, those empty neighboring blocks without shade are adjacent neighboring blocks, and those with shade are non-adjacent neighboring blocks. In the above method, the coefficients of the proposed motion compensation filter are always derived from the reconstructed samples adjacent to the current coded block (i.e., the direct top and left neighboring blocks). Such a scheme may be effective when the current block is highly correlated with its adjacent spatial neighbors. However, in real-world encoding / decoding scenarios, due to the presence of encoding / decoding noise (e.g., encoding / decoding noise caused by quantization / dequantization and block artifacts introduced during the motion compensation stage), the current block may be more correlated with samples in the reconstructed region that are not adjacent to the current block. Based on this consideration, an adaptive motion compensation filtering scheme based on non-adjacent neighbors is proposed in this section. Using this scheme, the coefficients of the adaptive motion compensation filter can be derived from samples in non-adjacent regions. Different methods can be applied to locate non-adjacent reconstructed samples to derive the filter coefficients. In one or more embodiments, non-adjacent neighboring blocks can be scanned from the left and top regions of the current block. The scan distance can be defined as the number of scan block sizes to the left or top of the current block.

[0246] As shown in Figure 17, multiple rows (columns) of non-adjacent neighboring blocks can be scanned at the top or left of the current block. The distances shown in Figure 17 represent the number of scan block sizes from each candidate location to the current block, with each scan block size representing a unit of distance. For example, a region to the left of the current block with a "distance 2" indicates that the candidate neighboring block located in that region is 2 scan block sizes away from the current block. Based on this pattern, different scan block sizes can be applied: in one approach, as shown in Figure 17A, non-adjacent neighboring blocks at each distance can have the same block size as the current block. Note that when this approach is applied, the granularity of the block scan is adaptively adjusted according to the segmentation granularity of the current block; that is, larger coded blocks have a greater opportunity to utilize more distant non-adjacent reconstructed samples to compute the coefficients of the adaptive filter.

[0247] In another approach, non-neighboring blocks that can be obtained for filter coefficient derivation can be defined based on fixed blocks (e.g., 4×4, 8×8).

[0248] In the third method, a combination approach can be used to define the scan pattern. For example, for small blocks, a fixed scan block size (Ws × Hs) can be applied, where Ws and Hs are the width and height of the fixed scan block size; otherwise, for large blocks, the scan block size is defined as the current block size. Specifically, let xStep and yStep indicate the width and height of the scan block size, with corresponding values ​​of xStep = max(Ws, width) and yStep = max(Hs, height), where width and height are the width and height of the current block.

[0249] To indicate the use of non-neighboring neighbors for filter derivation, the spatial candidate list can be formed by including both neighboring neighbors (i.e., the directly top spatial neighbor reconstructed samples and the left spatial neighbor reconstructed samples) and non-neighboring neighbor blocks. For signaling, a flag can be sent first to indicate the use of the spatial candidate list. And, when this flag equals one (i.e., candidate list selected), an index can be sent from the encoder to the decoder to specify which spatial candidate is selected to derive the filter coefficients. Furthermore, a template-based reordering scheme can be applied to reduce the signaling overhead of the index. Specifically, this scheme calculates the distortion between the template samples of the current block (i.e., the reconstructed neighboring samples above and to the left of the current block) and the predicted samples of the template samples generated using the motion and filter coefficients of each candidate in the list, and uses this distortion to sort the spatial candidates in the list in ascending order based on the template distortion; then, the corresponding index of the selected candidate is sent by signaling based on the reordered list with an appropriate context model.

[0250] In another approach, it is proposed to directly add new non-neighboring neighbors and neighboring neighbors associated with their corresponding adaptive filter coefficients to the existing merge candidate list (i.e., spatial MVP, TMVP, non-neighboring MVP, HMVP, pairwise merge candidates, and zero candidates). With this scheme, the use of the proposed non-neighboring / neighboring neighbors for adaptive filtering can be indicated by the index of the existing merge pattern. Similarly, a template-based reordering scheme can be applied to reduce the signaling overhead of the merge index.

[0251] In one embodiment of this disclosure, all inter-frame blocks are allowed to be used as references for the proposed filter coefficient inheritance. In another embodiment of this disclosure, inter-frame blocks encoded and decoded using SbTMVP mode and GPM mode are excluded from being used as references in the proposed filter coefficient inheritance scheme.

[0252] Alternatively, in some examples, it is proposed to apply the proposed non-neighboring spatial neighbors to existing LIC designs, where the proposed adaptive motion compensation filter degenerates into a 2-tap filter (i.e., one scaling and one offset). Specifically, based on the motion information of the current block (one-way or two-way prediction), the method uses the motion information to generate corresponding prediction signals for selected non-neighboring blocks, which are then used to derive corresponding LIC parameters by minimizing the difference between the reconstructed samples of the non-neighboring blocks and their corresponding predictions.

[0253] In all the methods described above, different approaches can be applied to generate the filter coefficients for an inter-frame block, which are then used to filter the predicted samples for future inter-frame blocks. In the first approach, a single set of filter coefficients is proposed for each inter-frame block. Specifically, when the block is unidirectionally predicted (i.e., predicted from a reference list L0 / L1), its filter coefficients are directly derived based on its reconstructed samples and its unidirectional predicted samples. When the block is bidirectionally predicted (i.e., predicted from both reference lists L0 and L1), the final bidirectional predicted samples are first generated based on a weighted average of the L0 and L1 predicted samples. A single set of filter coefficients can then be derived by minimizing the difference between the block's reconstructed samples and the bidirectional predicted samples. Similarly, once the current block references the filter coefficients of a non-adjacent / adjacent block, predicted samples (either unidirectional or bidirectional based on the motion of the reference block) are first generated, and then further modified based on the corresponding filter coefficients.

[0254] In another approach, two sets of filter coefficients are generated for each inter-frame block of bidirectional prediction, one set for L0 and the other for L1. Furthermore, various derivation schemes, such as those proposed in "Adaptive Bidirectional Prediction Filter Derivation Based on Unidirectional Samples" and "Adaptive Bidirectional Prediction Filter Derivation Based on Recursive Unidirectional Samples," can be applied, aiming to minimize the distortion between the reconstructed samples of the L0 and L1 prediction samples of the current bidirectional prediction block.

[0255] In another approach, the coefficients of the adaptive filtering applied when generating the prediction samples of inter-frame blocks are directly stored. Therefore, when applying this method, no further derivation of filter coefficients is needed after reconstruction, as the filter coefficients have already been calculated when generating the prediction samples of the blocks. However, the cost of this method is that only blocks to which adaptive filtering is applied are allowed to be used as references for the prediction filtering processing of the current block, rather than all previous inter-frame blocks. This may reduce the overall usage of the proposed scheme, thus degrading the overall encoding / decoding performance.

[0256] In the above discussion, the adaptive filter coefficients of adjacent / non-adjacent inter-frame CUs are derived using predicted and reconstructed samples. As previously shown, the filter coefficients are derived based on linear regression aimed at minimizing the energy difference between predicted and reconstructed samples. In practice, to achieve various performance / complexity tradeoffs, different methods can be applied to select the sample locations within an inter-frame CU for deriving its adaptive filter. As shown in Figure 24A, one approach proposes considering all predicted and reconstructed samples within an inter-frame CU to derive its adaptive filter coefficients. In another approach, a subsampling scheme can be applied to select the corresponding sample locations (e.g., every Nth sample in the horizontal direction and every Mth sample in the vertical direction, as shown in Figure 24B), and then only the predicted and reconstructed samples at the selected locations are used for filter derivation. In yet another approach, as shown in Figure 24C, it is proposed to use only the predicted and reconstructed samples at the bottom and right boundaries for filter derivation. To switch between different filter derivation methods, a signaling syntax element can be used to indicate which set of filter coefficients to use. For example, in a particular example, it is proposed to allow the deriving of filter coefficients from all predicted / reconstructed samples of a neighboring / non-neighboring inter-frame CU (as shown in Figure 24A) or from the predicted / reconstructed samples at the bottom and right boundaries of a neighboring / non-neighboring inter-frame CU (as shown in Figure 24C). Accordingly, when the proposed adaptive filter is selected (e.g., via a merging mode), an additional flag is further signaled to indicate which sample locations within the selected neighboring / non-neighboring blocks are used to derive the corresponding filter coefficients.

[0257] In all the methods described above, the filter coefficients of adjacent / non-adjacent neighboring blocks are directly used for the current CU. Since the correlation between predicted and reconstructed samples from neighboring CUs can differ significantly from that between predicted and reconstructed samples from the current CU, this approach may not always be optimal. Therefore, to further improve its performance, this paper proposes an adaptive offset adjustment scheme to further apply to the proposed adaptive motion compensation filtering scheme. Specifically, it is assumed that the inherited filter coefficients are... The offset of adjacent / non-adjacent neighbors is The offset of the current CU's adaptive filter is calculated as follows:

[0258]

[0259] in, and These are the reconstructed and predicted samples from the current CU template (i.e., the direct neighboring locations), where N is the total number of samples in the template. and These are the offsets before and after adjustment. One approach proposes that the aforementioned offset adjustment be consistently applied in the proposed adjacent / non-adjacent adaptive filtering scheme; that is, the offset of the filter coefficients of the selected neighboring blocks is always adjusted before applying the offset of the filter coefficients of the selected neighboring blocks to the current CU. In another approach, when adaptive filtering is applied, a flag is sent to indicate whether offset adjustment is needed. When the flag is false, it means that the filter coefficients inherited from the adjacent / non-adjacent blocks are directly applied to the current CU; otherwise, when the flag is true, it means that the offset of the filter coefficients is adjusted before applying the filter to the motion compensation samples of the current CU.

[0260] Adaptive motion compensation filtering based on historical filter coefficients In the aforementioned non-neighboring neighbor-based scheme, filter coefficients are derived from reconstructed regions far from the current block, requiring additional on-chip memory to store those non-neighboring reconstructed samples. This is relatively expensive for practical hardware codec implementations. Therefore, to reduce implementation costs, a history-based adaptive motion compensation filtering method is proposed. In this method, the filter coefficients of a previously encoded block are stored in a table and can be used to filter motion compensation samples for future blocks. In some embodiments, this table can be a list of candidate filters. Tables with multiple sets of filter coefficients can be maintained and synchronized during encoding and decoding. Whenever an inter-frame block is encoded or decoded, a set of filter coefficients can be derived based on its reconstructed samples and its predicted samples, and then added as a new candidate to the last entry of the table. To maintain table size, a first-in-first-out (FIFO) rule can be used, where a redundancy check can be applied to check if there is a candidate in the table that is identical to the new candidate. If so, the identical candidate is removed from the table, all other candidates are moved forward, and the new candidate is added at the last entry. If the table is full and there is no identical candidate in the table, the first candidate is removed from the table, and the new candidate is added at the end. Then, candidate groups of filter coefficients can be selected for filtering motion-compensated samples of future coding blocks. For signaling, when selecting historically based filter coefficient derivations, an index can be signaled to indicate which candidate group in the table will be used to derive the filter coefficients for the current block. In another embodiment, to reduce the number of filter coefficient derivations, it is proposed to include only the filter coefficients of the coding blocks for which adaptive motion compensation filtering is selected in the table.

[0261] Alternatively, in some examples, it is proposed to apply the proposed history-based filter derivation scheme to existing LIC designs, where the proposed adaptive motion compensation filter degenerates into a 2-tap filter. Specifically, in this case, each candidate in the table consists of two parameters, namely, a scaling and an offset, which can be selected by an inter-frame coding block to adjust its prediction samples.

[0262] In another approach, new candidates associated with historical filter coefficients are directly added to the existing merge candidate list (i.e., spatial MVP, TMVP, non-adjacent MVP, HMVP, pairwise merge candidates, and zero candidates). This scheme allows the use of proposed historical filter coefficients to be indicated by an index of the existing merge patterns. Similar to the previous approach, a template-based reordering scheme can be applied to reduce the signaling overhead of the merge index.

[0263] Combination of Adaptive Motion Compensation Filtering and OBMC This section provides a method for applying the proposed adaptive motion compensation filtering method to the OBMC process. Specifically, in some example methods, it is proposed that when performing the OBMC process for the current block, in addition to considering the motion vectors of neighboring blocks in their corresponding motion compensation prediction samples, the LIC parameters of each neighboring block are also considered in their corresponding motion compensation prediction samples. For ease of description, the proposed method is illustrated below using conventional inter-frame prediction without sub-block segmentation as an example. For example, after combining the prediction signal of the current CU with multiple prediction signals based on the MV of its spatial neighbors, let... Representing coordinates Mixed prediction samples at the location. Represents the coordinates of the current CU Predicted sample points at the location; and This represents a predicted sample point at the same location as the current CU, but this predicted sample point is generated using the MVs of the CU's left and right neighbors, respectively. In some embodiments, as shown in equation (29), It can be , and The weighted average.

[0264]

[0265] Additionally, for illustrative purposes, it is assumed that an adaptive motion compensation filter is applied to the current block and its top and left neighbors in space, and that the applied filter is a single-tap filter (i.e., one scaling factor and one offset), where the filter coefficients... and Used for the current block, and Used for top adjacent blocks and For the left neighboring block. The proposed scheme first generates prediction samples for the current block, as shown below:

[0266] in, This uses the original predicted samples of the current block's motion vector without applying filtering. Then, the boundary predicted samples of the current CU are updated using the MVs of the top and left causal neighbors of the current CU. First, the top neighbor block of the current block is checked. If the top neighbor block is an inter-frame block, its MVs and filter coefficients (i.e., and The signal will be assigned to the current block to generate a prediction signal at the same position in the current block. .

[0267]

[0268] in, This is the original predicted sample of the current block using the motion vector of the top neighboring block without applying filtering. Then, the same process is followed to generate corresponding predicted samples based on the motion vector of the left neighboring block and the LIC parameters.

[0269] in, The original predicted samples of the current block are obtained using the motion vectors of the left neighboring block without applying filtering. Finally, the three predicted signals are combined according to a template-based OBMC mixing process (as described in the section "Overlapping Block Motion Compensation") to generate the final predicted samples of the current block.

[0270] When encoding and decoding the current block using a sub-block pattern (e.g., affine, ATMVP, and DMVR), the proposed motion-compensated filtering-based OBMC can also be applied to the internal OBMC of sub-blocks within the current CU. Specifically, when applying this scheme, the same filtering process shown in equations (29) to (31) can be applied to generate corresponding prediction samples for each sub-block using the top, left, bottom, and right neighboring sub-blocks of each sub-block. However, instead of the LIC parameters of the spatially neighboring blocks, the filter coefficients of the current CU will always be applied to the prediction sample derivation of the internal OBMC process.

[0271] To achieve different complexity / performance tradeoffs, this paper proposes two methods when applying the proposed motion-compensated filter OBMC. In one method, the filter-based OBMC is applied only to prediction samples on the CU boundary, but not to prediction samples of sub-blocks within the CU (i.e., internal OBMC). In this case, for internal OBMC, only the neighboring motion vectors of each sub-block's neighboring blocks are considered to generate its OBMC prediction samples. In the other method, the filter-based OBMC is applied to both the prediction samples on the CU boundary and the prediction samples on the sub-block boundaries of the sub-blocks within the CU.

[0272] Additionally, in equations (30) and (31), adaptive filter parameters of neighboring blocks are applied to generate corresponding prediction samples for the OBMC process of the current block. Since the filters for neighboring blocks are derived, such a design can lead to increased complexity in the hardware / software implementation. To reduce complexity, in one embodiment of this disclosure, instead of using the filter parameters of neighboring blocks, it is proposed to use the filter parameters of the current block when generating OBMC prediction samples from neighboring blocks. Specifically, when encoding / decoding the current block with adaptive motion compensation filtering enabled, the filter coefficients of the current block are applied to modify the OBMC prediction samples generated from the motion information of each neighboring block. Otherwise, if adaptive motion compensation filtering is not applied to the current block, it is not applied to the OBMC process for generating prediction samples of any neighboring block, even if the neighboring block itself has adaptive motion compensation filtering applied to it.

[0273] Furthermore, in a specific example, it is proposed to apply the above method to an existing LIC design. Specifically, for all the methods discussed above, the adaptive motion compensation filtering process degenerates into a 2-tap filter, i.e., a scaling factor plus an offset.

[0274] Combination of adaptive motion compensation filtering and template matching-based inter-frame tools As discussed in the "Background Art" section, several template matching-based techniques have been introduced in ECM to reduce the overhead of merging patterns. For example, in ARMC, candidates in the initial merging candidate list are subgrouped, and candidates in each subgroup are reordered based on the cost between the template sample and its corresponding predicted sample (i.e., the reference sample). In this way, candidates with better MV (i.e., smaller template cost) are associated with smaller merging indices. Similarly, in MMVD patterns, template cost is used to reorder all possible MMVD refinement positions, and only the encoder / decoder is allowed to select the first few positions after reordering. In this disclosure, a method is proposed to apply the proposed adaptive motion compensation filtering to the cost calculation of template matching-based schemes.

[0275] In the first approach, when applying adaptive motion compensation filtering to a merge candidate, it is proposed that the adaptive motion compensation filtering is always bypassed when calculating its template cost. However, if a candidate is selected (e.g., as indicated by the merge index), the adaptive motion compensation filtering is still applied to generate the predicted samples of the block. To illustrate the above approach, as shown in Figure 21, assume there are L merge candidates, i.e., M0, M1, ..., M L-1 Furthermore, in the non-general case, assume that the merge candidate and M are... i It is M j An adaptive motion-compensated filter is applied, while other merging candidates are not. This method bypasses the adaptive motion-compensated filter when calculating the difference between template samples and corresponding template prediction samples using the motion of L merging candidates during the template-based reordering process. However, the choice depends on whether M is ultimately selected. i Or M j Adaptive motion compensation filtering can still be applied to generate the final predicted samples for the block.

[0276] In the second method, adaptive motion compensation filtering is proposed to be applied to both the calculation of template cost and the generation of predicted sample points for the block. As shown in Figure 22, unlike the first method, in calculating M... i and M j Before the corresponding cost, adaptive motion compensation filtering is applied to adjust M. i and M jThe template prediction samples. In addition, during the reordering process, when a merge candidate is bidirectionally predicted, different adaptive filtering methods can be applied to adjust the template prediction samples. In one method (method #1), it is proposed to use the method discussed in the section "Adaptive Bidirectional Prediction Filtering Based on Template Bidirectional Prediction Samples" to generate the template prediction samples for each bidirectional prediction merge candidate. Specifically, in the proposed scheme, the bidirectional prediction samples of the template samples are first generated based on the L0 MV and L1 MV of the merge candidate; then, an adaptive filter is derived and applied to the bidirectional prediction samples of the template samples, as shown in (14). In the second method (method #2), it is proposed to use the method discussed in the section "Adaptive Bidirectional Prediction Filtering Based on Template Unidirectional Samples" to generate the template prediction samples for each bidirectional prediction merge candidate. Specifically, in this method, template prediction samples in L0 and L1 are first generated using the MV in L0 and L1 respectively; then, two adaptive filters are derived and applied to the L0 and L1 prediction samples of the template in a one-sided manner, and then combined to generate the final prediction samples of the template, as shown in (18) and (19). In the third method (method #3), a method is proposed to generate template prediction samples for each bidirectional prediction merging candidate using the method discussed in the section "Adaptive Bidirectional Prediction Filtering Based on Recursive One-Way Filtering". Specifically, with such a scheme, template prediction samples in L0 and L1 are first generated using the MV in L0 and L1 respectively; then, two filters are iteratively derived to apply to the prediction samples of the two one-way predictions of the template, and the prediction samples of the two one-way predictions are then combined to generate the final prediction samples of the template. In fact, different adaptive filtering schemes can be applied to different template matching schemes, which may lead to different coding efficiency / complexity trade-offs. In a specific example, it is proposed to apply method #3 to both ARMC mode and regular MMVD mode, and to apply method #2 to affine MMVD mode.

[0277] Furthermore, in specific examples, it is proposed to apply the above methods to existing LIC designs. Specifically, for all the methods discussed above, the adaptive motion compensation filtering process degenerates into a 2-tap filter, i.e., a scaling factor plus an offset.

[0278] Combination of adaptive motion compensation filtering and AMVP merging modeAs previously described, in the AMVP-merge mode, merge candidates corresponding to a given AMVP part are implicitly determined by minimizing the bilateral matching cost between the AMVP part and the merge part. For each merge candidate in the merge candidate list, the bilateral matching cost is calculated using the merge candidate MV and the AMVP MV. The merge candidate with the minimum cost is selected for the AMVP MV. Furthermore, when bilateral matching is enabled, it is refined and applied to the coding block starting with the selected merge candidate MV and the AMVP MV; otherwise, if template matching is enabled, it is refined and applied to the coding block starting with the selected merge candidate MV and the AMVP MV. As previously analyzed, the proposed adaptive motion compensation filter can compensate for illumination variations between the predicted block and the current block. Therefore, when applying the adaptive motion compensation filter to a merge candidate, bilateral matching, which aims to measure the average illumination difference between two blocks, may not be effective in evaluating the validity of the merge candidate. Based on this consideration, in one embodiment of this disclosure, when a given AMVP MV has one or more merge candidates in its corresponding merge candidate list, it is proposed to utilize the template matching cost to select the merge candidate associated with the AMVP MV. Specifically, in this case, for each merge candidate in the merge candidate list, the template matching cost is calculated using the merge candidate MV and the AMVP MV. Additionally, for merge candidates associated with adaptive motion compensation filtering, the filtering process is applied when calculating the corresponding template cost. Figure 23 provides an example illustrating this approach. In another embodiment, it is proposed that the bilateral matching cost is always still applied to select merge candidates for each AMVP MV, regardless of whether any merge candidates are associated with adaptive motion compensation filtering.

[0279] Furthermore, in specific examples, it is proposed to apply the above methods to existing LIC designs. Specifically, for all the methods discussed above, the adaptive motion compensation filtering process degenerates into a 2-tap filter, i.e., a scaling factor plus an offset.

[0280] Combination of Adaptive Motion Compensation Filtering and CIIPAs discussed above, in existing CIIP designs, inter-frame prediction samples are always generated solely based on the corresponding motion before mixing inter-frame prediction samples and intra-frame prediction samples. In one embodiment of this disclosure, to further improve encoding and decoding performance, it is proposed to also apply the proposed adaptive motion compensation filtering process to generate the corresponding CIIP inter-frame prediction samples. Specifically, when the merging candidate for the inter-frame portion is associated with adaptive motion compensation filtering (e.g., through merging inheritance), adaptive motion filtering is applied to modify the motion compensation prediction samples generated based on their MV. In the above method, when adaptive motion compensation filtering is enabled for a current block, it is always invoked to generate CIIP inter-frame prediction samples. Such a design may introduce non-negligible encoding / decoding complexity. To control computational complexity, in another embodiment of this disclosure, it is proposed to enable adaptive motion compensation filtering for a CIIP block only when the current image is a low-latency image, wherein the POC of all reference images in the low-latency image is no greater than the POC of the current image. In another embodiment of this disclosure, it is proposed that adaptive motion compensation filtering be enabled for CIIP blocks only when the current image is a low-latency image and the POC distance between the current image and the first reference image in list L0 is equal to 1.

[0281] Furthermore, in a specific example, it is proposed to apply the above method to an existing LIC design. Specifically, for all the methods discussed above, the adaptive motion compensation filtering process degenerates into a 2-tap filter, i.e., a scaling factor plus an offset.

[0282] Figure 25 illustrates a computing environment (or computing device) 2510 coupled to a user interface 2560. The computing environment 2510 may be part of a data processing server. In some embodiments, the computing device 2510 may perform any of the various methods or processes described above (such as encoding / decoding methods or processes) according to various examples of this disclosure. The computing environment 2510 may include a processor 2520, a memory 2540, and an I / O interface 2550.

[0283] Processor 2520 typically controls the overall operation of computing environment 2510, such as operations associated with display, data acquisition, data communication, and image processing. Processor 2520 may include one or more processors for executing instructions to perform all or some of the steps in the methods described above. Furthermore, processor 2520 may include one or more modules that facilitate interaction between processor 2520 and other components. The processor may be a central processing unit (CPU), microprocessor, microcontroller, graphics processing unit (GPU), etc.

[0284] Memory 2540 is configured to store various types of data to support the operation of computing environment 2510. Memory 2540 may include predefined software 2542. Examples of such data include instructions for any application or method operating on computing environment 2510, video datasets, image data, etc. Memory 2540 can be implemented using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0285] I / O interface 2550 provides an interface between processor 2520 and peripheral interface modules (such as keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 2550 can be coupled to encoders and decoders.

[0286] In an embodiment, a non-transitory computer-readable storage medium is also provided, including, for example, a plurality of programs in memory 2530 and / or a bitstream generated by the above-described encoding method or a bitstream to be decoded by the above-described decoding method. The plurality of programs can be executed by processor 2520 in computing environment 2510 to perform the above-described methods. In one example, the plurality of programs can be executed by processor 2520 in computing environment 2510 to receive (e.g., from video encoder 20 in FIG. 1G) a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by processor 2520 in computing environment 2510 to perform the above-described decoding method based on the received bitstream or data stream. In another example, the plurality of programs can be executed by processor 2520 in computing environment 2510 to perform the above-described encoding method to encode video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bitstream or data stream, and can also be executed by processor 2520 in computing environment 2510 to send the bitstream or data stream (e.g., to video decoder 30 in FIG. 2B). Alternatively, a non-transitory computer-readable storage medium may store a bitstream or data stream comprising encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.) generated by an encoder (e.g., video encoder 20 in FIG. 1G) using, for example, the encoding method described above, for use by a decoder (e.g., video decoder 30 in FIG. 2B) when decoding video data. The non-transitory computer-readable storage medium may be, for example, ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0287] In one embodiment, a bitstream generated by the above-described encoding method or a bitstream to be decoded by the above-described decoding method is provided. In another embodiment, a bitstream is provided that includes encoded video information generated by the above-described encoding method or encoded video information to be decoded by the above-described decoding method.

[0288] In one embodiment, a computing device is also provided, comprising: one or more processors (e.g., processor 2520); and a non-transitory computer-readable storage medium or memory 2530 therein storing a plurality of programs executable by the one or more processors, wherein the one or more processors are configured to perform the methods described above when executing the plurality of programs.

[0289] In one embodiment, a computer program product is also provided, having instructions for storing or transmitting a bitstream, the bitstream including encoded video information generated by the encoding method described above or encoded video information to be decoded by the decoding method described above. In another embodiment, a computer program product comprising multiple programs is also provided, the computer program product being, for example, stored in memory 2530, and executable by processor 2520 in computing environment 2510 to perform the methods described above. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0290] In an embodiment, the computing environment 2510 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the methods described above.

[0291] In one embodiment, a method for storing a bitstream is also provided, comprising: storing the bitstream on a digital storage medium, wherein the bitstream includes encoded video information generated by the above-described encoding method or encoded video information to be decoded by the above-described decoding method.

[0292] In one embodiment, a method for transmitting a bitstream generated by the encoder described above is also provided. In another embodiment, a method for receiving a bitstream to be decoded by the decoder described above is also provided.

[0293] Figure 26 is a flowchart illustrating a method for video decoding according to an example of the present disclosure. The method can be implemented by a decoder to decode inter-frame coded blocks. In step 2601, the method includes: determining, by the decoder, non-adjacent neighbor blocks of the current inter-frame coded block, wherein the non-adjacent neighbor blocks comprise a plurality of reconstructed samples that are not adjacent to the current inter-frame coded block. Due to the presence of encoding / decoding noise, the current inter-frame coded block may be more correlated with samples in reconstructed regions that are not adjacent to the current block. Based on this consideration, an adaptive motion compensation filtering scheme can be performed based on reconstructed samples that are not adjacent to the current inter-frame coded block. In some embodiments, non-adjacent neighbor blocks can be scanned from the left and top regions of the current inter-frame coded block. In step 2602, the method includes: obtaining, by the decoder, a plurality of predicted samples of the non-adjacent neighbor blocks based on the motion vectors of the non-adjacent neighbor blocks. In step 2603, the method includes: obtaining a filter by the decoder based on the plurality of predicted samples and the plurality of reconstructed samples. In some embodiments, non-adjacent neighbor blocks are predicted unidirectionally (i.e., predicted from a reference list L0 / L1), and their filter coefficients are directly derived based on their reconstructed samples and their unidirectional predicted samples. In some other embodiments, non-adjacent neighbor blocks are predicted bidirectionally (i.e., predicted from both reference lists L0 and L1), and the final bidirectional predicted samples are first generated based on a weighted average of L0 and L1 predicted samples. In step 2604, the current predicted block is obtained by the decoder based on the motion vector of the current inter-coding block. In step 2605, the filtered predicted block is obtained by the decoder based on the filter and the current predicted block. The current predicted block of the current inter-coding block is modified according to the filter coefficients of the filter derived based on multiple predicted samples and multiple reconstructed samples.

[0294] In some examples, multiple predicted samples of non-adjacent neighboring blocks are obtained based on the motion vectors of non-adjacent neighboring blocks, including obtaining multiple predicted samples of non-adjacent neighboring blocks from reference lists L0 or L1.

[0295] In some examples, multiple predicted samples of non-adjacent neighboring blocks are obtained based on the motion vectors of non-adjacent neighboring blocks, including: multiple first predicted samples of non-adjacent neighboring blocks obtained from reference list L0; and multiple second predicted samples of non-adjacent neighboring blocks obtained from reference list L1.

[0296] In some examples, obtaining a filter based on multiple predicted samples and multiple reconstructed samples includes: obtaining multiple predicted samples of non-adjacent neighboring blocks based on a first predicted sample and a second prediction of multiple reconstructed samples; and obtaining a filter by minimizing the difference between the multiple predicted samples of non-adjacent neighboring blocks and the multiple reconstructed samples. Obtaining a current prediction block based on the motion vector of the current inter-coding block includes: obtaining a first current prediction block from reference list L0 based on the motion vector of the current inter-coding block; obtaining a second current prediction block from reference list L1 based on the motion vector of the current inter-coding block; and obtaining a current prediction block based on the first and second current prediction blocks. Obtaining a filtered prediction block based on the filter and the current prediction block includes: obtaining a filtered prediction block by applying a filter to the current prediction block.

[0297] In some examples, a filter is obtained based on multiple predicted samples and multiple reconstructed samples, including: obtaining a first filter by minimizing the difference between multiple first predicted samples and multiple reconstructed samples; obtaining a second filter by minimizing the difference between multiple second predicted samples and multiple reconstructed samples; wherein, a filtered prediction block is obtained based on the filter and the current prediction block, including: obtaining a first current prediction block and a second current prediction block from reference list L1 and list L0 respectively based on the motion vector of the current inter-frame coding block; obtaining a first filtered prediction block by applying the first filter to the first current prediction block; obtaining a second filtered prediction block by applying the second filter to the second current prediction block; and obtaining a filtered prediction block based on the first filtered prediction block and the second filtered prediction block.

[0298] In some examples, a filter is obtained based on multiple predicted samples and multiple reconstructed samples, including: calculating multiple target reconstructed samples based on the multiple reconstructed samples and previous filtered predictions of the multiple reconstructed samples; obtaining the coefficients of the current filter by minimizing the difference between the current prediction of the multiple reconstructed samples and the multiple target reconstructed samples; and calculating the filtered prediction of the multiple reconstructed samples by applying the current filter to the current prediction.

[0299] In some examples, the method further includes: in response to receiving a first index sent by the encoder via a signal, determining, via the decoder, non-adjacent neighboring blocks in the candidate list based on the first index.

[0300] In some examples, the candidate list is reordered based on the distortion of multiple neighboring reconstructed samples of the current inter-frame coding block by using motion information and filters associated with each non-adjacent neighboring block candidate.

[0301] In some examples, the candidate list is generated by merging non-neighboring neighbor block candidates associated with the corresponding adaptive filter coefficients into an existing merged candidate list.

[0302] In some examples, the existing merge candidate list includes candidate lists for the following items: Spatial Motion Vector Prediction (MVP), Temporal Motion Vector Prediction (TMVP), Non-adjacent MVP, History-Based Motion Vector Prediction (HMVP), Paired Merge Candidates, or Zero Candidates.

[0303] In some examples, the method further includes: in response to receiving a second index signaled by the encoder, determining, via the decoder, to use the candidate list to identify non-adjacent neighboring blocks in the candidate list.

[0304] In some examples, non-neighboring neighbor block candidates encoded using the sub-block-based temporal motion vector prediction (SbTMVP) mode or the geometric segmentation mode (GPM) mode are excluded from the candidate list.

[0305] Figure 27 is a flowchart illustrating a method for video decoding according to an example of the present disclosure. The method can be implemented by a decoder to decode inter-frame coded blocks. In step 2701, the method includes: obtaining, by the decoder, filter coefficients of a filter applied to a previous decoded block for the current inter-frame coded block. The filter coefficients of the previous decoded block are used to perform adaptive filtering on the current inter-frame coded block. In step 2702, the method includes: obtaining a current prediction block based on the motion vector of the current inter-frame coded block by the decoder. In step 2703, the method includes: obtaining a filtered prediction block based on the filter and the current prediction block by the decoder, wherein the filter is derived based on a plurality of predicted samples from non-adjacent neighbor blocks of the previous decoded block, and non-adjacent neighbor blocks comprising a plurality of reconstructed samples not adjacent to the previous decoded block. The filter applied to the current prediction block is a filter derived when adaptive filtering is applied to the previous decoded block, such that it is not necessary to derive filter coefficients after reconstruction for the current inter-frame coded block.

[0306] Figure 28 is a flowchart illustrating a method for video decoding according to an example of this disclosure. The method can be implemented by a decoder to decode inter-frame coded blocks. In step 2801, the method includes obtaining a filter by the decoder based on a plurality of reconstructed samples of the current inter-frame coded block and a plurality of predicted samples of the plurality of reconstructed samples. In step 2802, the method includes obtaining a current predicted block by the decoder based on the motion vector of the current inter-frame coded block. In step 2803, the method includes obtaining a filtered predicted block by the decoder based on the filter and the current predicted block, wherein the plurality of reconstructed samples are located in an upper or left region adjacent to the current inter-frame coded block. Due to the different correlations between the current inter-frame coded block and its top / left neighbors, two additional adaptive filtering modes are proposed: in a first mode (referred to as the top-only filtering mode), filter coefficients are derived using only the neighboring samples above the current inter-frame coded block (i.e., the top template); in a second mode (referred to as the left-only filtering mode), filter coefficients of the current block are derived using only the neighboring samples to the left of the current inter-frame coded block (i.e., the left template).

[0307] In some examples, the method further includes: in response to receiving a first flag indicating whether the multiple reconstructed samples are in the upper or left region of the current inter-frame coding block, determining the multiple reconstructed samples by the decoder based on the first flag.

[0308] In some examples, the method further includes receiving a second flag via a decoder before receiving the first flag, the second flag indicating that multiple reconstructed samples are in either the upper or left region of the current inter-frame coded block.

[0309] In some examples, the current inter-frame coded block will be decoded according to the Advanced Motion Vector Prediction (AMVP) mode.

[0310] In some examples, the current prediction block is obtained by the decoder based on the motion vector of the current inter-coded block, including obtaining the current prediction block from reference list L0 or L1 based on the motion vector of the current inter-coded block.

[0311] In some examples, the filter coefficients include scaling factors and offsets.

[0312] Figure 29 is a flowchart illustrating a method for video decoding according to an example of the present disclosure. This method can be implemented by a decoder to decode inter-frame coded blocks. In step 2901, the method includes: determining, by the decoder, adjacent or non-adjacent neighbor blocks of the current inter-frame coded block, wherein the adjacent or non-adjacent neighbor blocks comprise a plurality of reconstructed samples adjacent to or not adjacent to the current inter-frame coded block. In step 2902, the method includes: obtaining, by the decoder, a plurality of predicted samples of partial reconstructed samples from the plurality of reconstructed samples. In some embodiments, to achieve various performance / complexity trade-offs, different methods can be applied to select partial reconstructed samples from the plurality of reconstructed samples for deriving an adaptive filter. In step 2903, the method includes: obtaining a filter by the decoder based on the plurality of predicted samples and the partial reconstructed samples from the plurality of reconstructed samples. For example, as shown in Figures 24A-24C, different subsampling schemes can be applied to select the corresponding sample positions of partial reconstructed samples from the plurality of reconstructed samples for deriving an adaptive filter. In step 2904, the method includes: obtaining a current prediction block based on the motion vector of the current inter-frame coded block using a decoder. In step 2905, the method includes: obtaining a filtered prediction block based on a filter and the current prediction block using a decoder.

[0313] In some examples, the partial reconstructed samples among multiple reconstructed samples include: reconstructed samples in the bottom boundary of adjacent or non-adjacent neighboring blocks; or, reconstructed samples in the right boundary of adjacent or non-adjacent neighboring blocks; or, reconstructed samples at sample locations selected based on the subsampling scheme in adjacent or non-adjacent neighboring blocks.

[0314] In some examples, the sample locations selected based on the subsampling scheme include: every Nth sample in the horizontal direction and every Mth sample in the vertical direction, where N and M are integers ≥ 2.

[0315] In some examples, the method further includes: receiving a first syntax element via a decoder, the first syntax element indicating whether the portion of the reconstructed samples among the plurality of reconstructed samples is used to derive the filter; or, receiving a second syntax element via a decoder, the second syntax element indicating the sample location of the portion of the reconstructed samples among the plurality of reconstructed samples.

[0316] Figure 30 is a flowchart illustrating a method for video decoding according to an example of the present disclosure. This method can be implemented by a decoder to decode inter-frame coded blocks. In step 3001, the method includes: determining, by the decoder, adjacent or non-adjacent neighbor blocks of the current inter-frame coded block, wherein the adjacent or non-adjacent neighbor blocks include a plurality of reconstructed samples adjacent to or not adjacent to the current inter-frame coded block. In step 3002, the method includes: obtaining, by the decoder, a plurality of predicted samples of the plurality of reconstructed samples. In step 3003, the method includes: obtaining, by the decoder, a filter based on the plurality of predicted samples and the plurality of reconstructed samples, wherein the coefficients of the filter include an offset. Since the correlation between the predicted samples and reconstructed samples of neighboring CUs may differ significantly from the correlation between the predicted samples and reconstructed samples of the current CU, an adaptive offset adjustment scheme can be further used to modify the coefficients of the filter. In step 3004, the method includes: calculating, by the decoder, an offset adjustment for the offset based on the filter coefficients, the plurality of predicted samples, the plurality of reconstructed samples, and the total number of samples in the adjacent or non-adjacent neighbor blocks. In step 3005, the method includes: adjusting the coefficients of the filter based on offset adjustment using a decoder. In step 3006, the method includes: obtaining a current prediction block based on the motion vector of the current inter-frame coded block using a decoder. In step 3007, the method includes: obtaining a filtered prediction block using a decoder, based on the filter adjusted according to offset adjustment and the current prediction block.

[0317] In some examples, the method also includes receiving a flag indicating that the filter offset needs to be adjusted via a decoder.

[0318] Figure 31 is a flowchart illustrating a method for video coding according to an example of the present disclosure. The method can be implemented by an encoder to encode inter-frame coded blocks. In step 3101, the method includes: determining, by the encoder, non-adjacent neighbor blocks of the current inter-frame coded block, wherein the non-adjacent neighbor blocks comprise a plurality of reconstructed samples that are not adjacent to the current inter-frame coded block. Due to the presence of encoding / decoding noise, the current inter-frame coded block may be more correlated with samples in reconstructed regions that are not adjacent to the current block. Based on this consideration, an adaptive motion compensation filtering scheme can be performed based on the reconstructed samples that are not adjacent to the current inter-frame coded block. In some embodiments, non-adjacent neighbor blocks can be scanned from the left and top regions of the current inter-frame coded block. In step 3102, the method includes: obtaining a plurality of predicted samples of the non-adjacent neighbor blocks based on the motion vectors of the non-adjacent neighbor blocks by the encoder. In step 3103, the method includes: obtaining a filter by the encoder based on the plurality of predicted samples and the plurality of reconstructed samples. In some embodiments, non-adjacent neighbor blocks are predicted unidirectionally (i.e., predicted from a reference list L0 / L1), and their filter coefficients are directly derived based on their reconstructed samples and their unidirectional predicted samples. In some other embodiments, non-adjacent neighbor blocks are predicted bidirectionally (i.e., predicted from both reference lists L0 and L1), and the final bidirectional predicted samples are first generated based on a weighted average of L0 and L1 predicted samples. In step 3104, the current predicted block is obtained by the encoder based on the motion vector of the current inter-coding block. In step 3105, the filtered predicted block is obtained by the encoder based on the filter and the current predicted block. The current predicted block of the current inter-coding block is modified according to the filter coefficients of the filter derived based on multiple predicted samples and multiple reconstructed samples.

[0319] In some examples, multiple predicted samples of non-adjacent neighboring blocks are obtained based on the motion vectors of non-adjacent neighboring blocks, including obtaining multiple predicted samples of non-adjacent neighboring blocks from reference lists L0 or L1.

[0320] In some examples, multiple predicted samples of non-adjacent neighboring blocks are obtained based on the motion vectors of non-adjacent neighboring blocks, including: multiple first predicted samples of non-adjacent neighboring blocks obtained from reference list L0; and multiple second predicted samples of non-adjacent neighboring blocks obtained from reference list L1.

[0321] In some examples, a filter is obtained based on multiple predicted samples and multiple reconstructed samples, including: obtaining multiple predicted samples of non-adjacent neighboring blocks based on a first and second predicted sample of the multiple reconstructed samples; and obtaining a filter by minimizing the difference between the multiple predicted samples of the non-adjacent neighboring blocks and the multiple reconstructed samples, wherein obtaining a current prediction block based on the motion vector of the current inter-coding block includes: obtaining a first current prediction block from reference list L0 based on the motion vector of the current inter-coding block; obtaining a second current prediction block from reference list L1 based on the motion vector of the current inter-coding block; and obtaining a current prediction block based on the first and second current prediction blocks, wherein obtaining a filtered prediction block based on the filter and the current prediction block includes: obtaining a filtered prediction block by applying a filter to the current prediction block.

[0322] In some examples, a filter is obtained based on multiple predicted samples and multiple reconstructed samples, including: obtaining a first filter by minimizing the difference between multiple first predicted samples and multiple reconstructed samples; obtaining a second filter by minimizing the difference between multiple second predicted samples and multiple reconstructed samples; wherein, a filtered prediction block is obtained based on the filter and the current prediction block, including: obtaining a first current prediction block and a second current prediction block from reference list L1 and list L0 respectively based on the motion vector of the current inter-frame coding block; obtaining a first filtered prediction block by applying the first filter to the first current prediction block; obtaining a second filtered prediction block by applying the second filter to the second current prediction block; and obtaining a filtered prediction block based on the first filtered prediction block and the second filtered prediction block.

[0323] In some examples, a filter is obtained based on multiple predicted samples and multiple reconstructed samples, including: calculating multiple target reconstructed samples based on the multiple reconstructed samples and previous filtered predictions of the multiple reconstructed samples; obtaining the coefficients of the current filter by minimizing the difference between the current prediction of the multiple reconstructed samples and the multiple target reconstructed samples; and calculating the filtered prediction of the multiple reconstructed samples by applying the current filter to the current prediction.

[0324] In some examples, the method further includes: sending a first index via an encoder to instruct the decoder to determine non-adjacent neighboring blocks in the candidate list based on the first index.

[0325] In some examples, the candidate list is reordered based on the distortion of multiple neighboring reconstructed samples of the current inter-frame coding block by using motion information and filters associated with each non-adjacent neighboring block candidate.

[0326] In some examples, the candidate list is generated by merging non-neighboring neighbor block candidates associated with the corresponding adaptive filter coefficients into an existing merged candidate list.

[0327] In some examples, the existing merge candidate list includes candidate lists for the following items: Spatial Motion Vector Prediction (MVP), Temporal Motion Vector Prediction (TMVP), Non-adjacent MVP, History-Based Motion Vector Prediction (HMVP), Paired Merge Candidates, or Zero Candidates.

[0328] In some examples, the method further includes: sending a second index via an encoder to instruct the decoder to use the candidate list to determine non-adjacent neighboring blocks in the candidate list.

[0329] In some examples, non-neighboring neighbor block candidates encoded using the sub-block-based temporal motion vector prediction (SbTMVP) mode or the geometric segmentation mode (GPM) mode are excluded from the candidate list.

[0330] Figure 32 is a flowchart illustrating a method for video coding according to an example of the present disclosure. The method can be implemented by an encoder to encode inter-frame coded blocks. In step 3201, the method includes: obtaining, by the encoder, filter coefficients of a filter applied to a previous coded block of the current inter-frame coded block. The filter coefficients of the previous decoded block are used to perform adaptive filtering on the current inter-frame coded block. In step 3202, the method includes: obtaining a current prediction block based on the motion vector of the current inter-frame coded block by the encoder. In step 3203, the method includes: obtaining a filtered prediction block based on the filter and the current prediction block by the encoder; wherein the filter is derived based on a plurality of predicted samples from non-adjacent neighbor blocks of the previous coded block, and non-adjacent neighbor blocks comprising a plurality of reconstructed samples not adjacent to the previous coded block. The filter applied to the current prediction block is a filter derived when adaptive filtering is applied to the previous decoded block, such that it is not necessary to derive filter coefficients after reconstruction for the current inter-frame coded block.

[0331] Figure 33 is a flowchart illustrating a method for video coding according to an example of the present disclosure. The method can be implemented by an encoder for encoding inter-frame coding blocks. In step 3351, the method includes: obtaining a filter based on a plurality of reconstructed samples of the current inter-frame coding block and a plurality of predicted samples of the plurality of reconstructed samples using the encoder. In step 3352, the method includes: obtaining a current prediction block based on the motion vector of the current inter-frame coding block using the encoder. In step 3353, the method includes: obtaining a filtered prediction block based on the filter and the current prediction block using the encoder, wherein the plurality of reconstructed samples are located in an upper or left region adjacent to the current inter-frame coding block. Due to the different correlations between the current inter-coding block and its top / left neighbors, two additional adaptive filtering modes are proposed: in the first mode (which is called the top-only filtering mode), the filter coefficients are derived using only the neighboring samples above the current inter-coding block (i.e., the top template); in the second mode (which is called the left-only filtering mode), the filter coefficients of the current block are derived using only the neighboring samples to the left of the current inter-coding block (i.e., the left template).

[0332] In some examples, the method further includes: sending a first flag via an encoder to indicate whether the multiple reconstructed samples are in the upper or left region of the current inter-frame coding block.

[0333] In some examples, the method further includes: sending a second flag via an encoder, the second flag indicating that a plurality of reconstructed samples are in either the upper or left region of the current inter-frame coded block.

[0334] In some examples, the current inter-frame coding block will be encoded according to the Advanced Motion Vector Prediction (AMVP) mode.

[0335] In some examples, the current prediction block is obtained by the encoder based on the motion vector of the current inter-coded block, including obtaining the current prediction block from reference list L0 or L1 based on the motion vector of the current inter-coded block.

[0336] In some examples, the filter coefficients include scaling factors and offsets.

[0337] Figure 34 is a flowchart illustrating a method for video coding according to an example of the present disclosure. The method can be implemented by an encoder to encode inter-frame coded blocks. In step 3401, the method includes: determining, by the encoder, adjacent or non-adjacent neighbor blocks of the current inter-frame coded block, wherein the adjacent or non-adjacent neighbor blocks comprise a plurality of reconstructed samples adjacent or non-adjacent to the current inter-frame coded block. In step 3402, the method includes: obtaining, by the encoder, a plurality of predicted samples of partial reconstructed samples from the plurality of reconstructed samples. In some embodiments, to achieve various performance / complexity trade-offs, different methods can be applied to select partial reconstructed samples from the plurality of reconstructed samples for deriving an adaptive filter. In step 3403, the method includes: obtaining a filter by the encoder based on the plurality of predicted samples and the partial reconstructed samples from the plurality of reconstructed samples. For example, as shown in Figures 24A-24C, different subsampling schemes can be applied to select the corresponding sample positions of partial reconstructed samples from the plurality of reconstructed samples for deriving an adaptive filter. In step 3404, the method includes: obtaining a current prediction block based on the motion vector of the current inter-frame coded block using an encoder. In step 3405, the method includes: obtaining a filtered prediction block based on a filter and the current prediction block using an encoder.

[0338] In some examples, the partial reconstructed samples among multiple reconstructed samples include: reconstructed samples in the bottom boundary of adjacent or non-adjacent neighboring blocks; or, reconstructed samples in the right boundary of adjacent or non-adjacent neighboring blocks; or, reconstructed samples at sample locations selected based on the subsampling scheme in adjacent or non-adjacent neighboring blocks.

[0339] In some examples, the sample locations selected based on the subsampling scheme include: every Nth sample in the horizontal direction and every Mth sample in the vertical direction, where N and M are integers ≥ 2.

[0340] In some examples, the method further includes: signaling a first syntax element via an encoder, the first syntax element indicating whether a portion of the reconstructed samples among the plurality of reconstructed samples are used to derive the filter; or signaling a second syntax element via an encoder, the second syntax element indicating the sample location of a portion of the reconstructed samples among the plurality of reconstructed samples.

[0341] Figure 35 is a flowchart illustrating a method for video coding according to an example of the present disclosure. The method can be implemented by an encoder to encode inter-frame coding blocks. In step 3501, the method includes: determining, by the encoder, adjacent or non-adjacent neighboring blocks of the current inter-frame coding block, wherein the adjacent or non-adjacent neighboring blocks include a plurality of reconstructed samples adjacent to or not adjacent to the current inter-frame coding block. In step 3502, the method includes: obtaining, by the encoder, a plurality of predicted samples of the plurality of reconstructed samples. In step 3503, the method includes: obtaining, by the encoder, a filter based on the plurality of predicted samples and the plurality of reconstructed samples, wherein the coefficients of the filter include an offset. Since the correlation between the predicted samples and reconstructed samples of neighboring CUs may differ significantly from the correlation between the predicted samples and reconstructed samples of the current CU, an adaptive offset adjustment scheme can be further used to modify the coefficients of the filter. In step 3504, the method includes: calculating, by the encoder, an offset adjustment for the offset based on the filter coefficients, the plurality of predicted samples, the plurality of reconstructed samples, and the total number of samples in the adjacent or non-adjacent neighboring blocks. In step 3505, the method includes: adjusting the coefficients of the filter based on offset adjustment using an encoder. In step 3506, the method includes: obtaining a current prediction block based on the motion vector of the current inter-frame coded block using an encoder. In step 3507, the method includes: obtaining a filtered prediction block using an encoder, based on the filter adjusted according to offset adjustment, and the current prediction block.

[0342] In some examples, the method also includes sending a flag via an encoder that indicates that the filter offset needs to be adjusted.

[0343] In some examples, an apparatus for video encoding and decoding is provided. The apparatus includes a processor 2520 and a memory 2540 configured to store instructions executable by the processor; wherein the processor, when executing the instructions, is configured to perform any of the methods shown in Figures 26-33.

[0344] In some other examples, a non-transitory computer-readable storage medium is provided having instructions stored therein. When executed by processor 2520, the instructions cause the processor to perform any of the methods shown in Figures 26-33. In one example, multiple programs may be executed by processor 2520 in computing environment 2510 to receive (e.g., from video encoder 20 in Figure 1G) a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and may also be executed by processor 2520 in computing environment 2510 to perform the above-described decoding method based on the received bitstream or data stream. In another example, multiple programs may be executed by processor 2520 in computing environment 2510 to perform the above-described encoding method to encode video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bitstream or data stream, and may also be executed by processor 2520 in computing environment 2510 to transmit the bitstream or data stream (e.g., to video decoder 30 in Figure 2B). Alternatively, the non-transitory computer-readable storage medium may store a bitstream or data stream comprising encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.) generated by an encoder (e.g., video encoder 20 in FIG. 1G) using, for example, the encoding method described above, for use by a decoder (e.g., video decoder 30 in FIG. 2B) when decoding video data. The non-transitory computer-readable storage medium may be, for example, ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0345] The description in this disclosure has been presented for illustrative purposes and is not intended to be exhaustive or limited to this disclosure. Many modifications, variations, and alternative embodiments will be apparent to those skilled in the art from the teachings presented in the foregoing description and the associated drawings.

[0346] Unless otherwise specifically stated, the order of steps in the method according to this disclosure is intended to be illustrative only, and the steps of the method according to this disclosure are not limited to the specific order described above, but may be changed according to actual circumstances. Furthermore, at least one step in the method according to this disclosure may be adjusted, combined, or omitted as needed.

[0347] The examples chosen and described are intended to explain the principles of this disclosure and to enable others skilled in the art to understand the various embodiments of this disclosure, and preferably to utilize the basic principles and various embodiments with various modifications suitable for the intended particular purpose. Therefore, it will be understood that the scope of this disclosure is not limited to the specific examples of the disclosed embodiments, and that modifications and other embodiments are intended to be included within the scope of this disclosure.

[0348] The methods described above can be implemented using an apparatus comprising one or more circuits, including application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components. The apparatus can be combined with other hardware or software components to perform the methods described above using the circuits. Each module, submodule, unit, or subunit disclosed above can be implemented at least partially using one or more circuits.

[0349] Other examples of this disclosure will be apparent to those skilled in the art upon consideration of the specification and practice disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include such deviations from this disclosure within the scope of known or customary practice in the art. The specification and examples are intended to be illustrative only.

[0350] It should be understood that this disclosure is not limited to the precise examples described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from its scope.

Claims

1. A method for video decoding, comprising: The decoder determines non-adjacent neighbor blocks of the current inter-frame coding block, wherein the non-adjacent neighbor blocks include multiple reconstructed samples that are not adjacent to the current inter-frame coding block; the decoder obtains multiple predicted samples of the non-adjacent neighbor blocks based on the motion vectors of the non-adjacent neighbor blocks; the decoder obtains a filter based on the multiple predicted samples and the multiple reconstructed samples; the decoder obtains a current prediction block based on the motion vectors of the current inter-frame coding block; and the decoder obtains a filtered prediction block based on the filter and the current prediction block.

2. The method according to claim 1, wherein, The step of obtaining multiple predicted samples of the non-adjacent neighboring blocks based on the motion vectors of the non-adjacent neighboring blocks includes: obtaining the multiple predicted samples of the non-adjacent neighboring blocks from a reference list L0 or L1.

3. The method according to claim 1, wherein, The step of obtaining multiple predicted samples of the non-adjacent neighboring blocks based on their motion vectors includes: obtaining multiple first predicted samples of the non-adjacent neighboring blocks from reference list L0; and obtaining multiple second predicted samples of the non-adjacent neighboring blocks from reference list L1.

4. The method according to claim 3, wherein, The step of obtaining a filter based on the plurality of predicted samples and the plurality of reconstructed samples includes: obtaining the plurality of predicted samples of the non-adjacent neighboring blocks based on the first predicted samples and the second prediction of the plurality of reconstructed samples; and obtaining a filter by minimizing the difference between the plurality of predicted samples of the non-adjacent neighboring blocks and the plurality of reconstructed samples, wherein the step of obtaining a current prediction block based on the motion vector of the current inter-frame coding block includes: obtaining a first current prediction block from reference list L0 based on the motion vector of the current inter-frame coding block; obtaining a second current prediction block from reference list L1 based on the motion vector of the current inter-frame coding block; and obtaining the current prediction block based on the first current prediction block and the second current prediction block, wherein the step of obtaining a filtered prediction block based on the filter and the current prediction block includes: obtaining the filtered prediction block by applying the filter to the current prediction block.

5. The method according to claim 3, wherein, The step of obtaining a filter based on the plurality of predicted samples and the plurality of reconstructed samples includes: obtaining a first filter by minimizing the difference between the plurality of first predicted samples and the plurality of reconstructed samples; and obtaining a second filter by minimizing the difference between the plurality of second predicted samples and the plurality of reconstructed samples; wherein, the step of obtaining a filtered prediction block based on the filter and the current prediction block includes: obtaining a first current prediction block and a second current prediction block from reference list L1 and list L0 respectively based on the motion vector of the current inter-frame coding block; obtaining a first filtered prediction block by applying the first filter to the first current prediction block; obtaining a second filtered prediction block by applying the second filter to the second current prediction block; and obtaining the filtered prediction block based on the first filtered prediction block and the second filtered prediction block.

6. The method according to claim 1, wherein, The filter is obtained based on the multiple predicted samples and the multiple reconstructed samples: multiple target reconstructed samples are calculated based on the multiple reconstructed samples and the previous filtered predictions of the multiple reconstructed samples; the coefficients of the current filter are obtained by minimizing the difference between the current prediction of the multiple reconstructed samples and the multiple target reconstructed samples. And by applying the current filter to the current prediction, the filtered prediction of the plurality of reconstructed samples is calculated.

7. The method according to claim 1, further comprising: In response to receiving a first index transmitted by a signal via the encoder, the decoder determines the non-adjacent neighboring blocks in the candidate list based on the first index.

8. The method according to claim 7, wherein, The candidate list is reordered based on the distortion of multiple neighboring reconstructed samples of the current inter-frame coding block calculated using motion information and filters associated with each non-adjacent neighboring block candidate.

9. The method according to claim 7, wherein, The candidate list is generated by merging non-neighboring block candidates associated with the corresponding adaptive filter coefficients into an existing merged candidate list.

10. The method according to claim 9, wherein, The existing merge candidate list includes the following candidate lists: Spatial Motion Vector Prediction MVP, Temporal Motion Vector Prediction TMVP, Non-adjacent MVP, History-based Motion Vector Prediction HMVP, Paired Merge Candidates, or Zero Candidates.

11. The method of claim 7, further comprising: In response to receiving a second index transmitted by the encoder via a signal, the decoder determines, using the candidate list, the non-adjacent neighbor block in the candidate list.

12. The method according to claim 7, wherein, Non-neighboring neighbor block candidates encoded using the sub-block-based temporal motion vector prediction SbTMVP mode or the geometric segmentation mode GPM mode are excluded from the candidate list.

13. A method for video decoding, comprising: The decoder obtains the filter coefficients of the filters applied to the previous decoded blocks in the current inter-frame coded block. The decoder obtains a current prediction block based on the motion vector of the current inter-frame coded block; and the decoder obtains a filtered prediction block based on the filter and the current prediction block; wherein the filter is derived based on multiple prediction samples of non-adjacent neighboring blocks of the previous decoded block and the non-adjacent neighboring blocks, the non-adjacent neighboring blocks including multiple reconstructed samples that are not adjacent to the previous decoded block.

14. A method for video decoding, comprising: The filter is obtained by the decoder based on multiple reconstructed samples of the current inter-frame coding block and multiple predicted samples of the multiple reconstructed samples; The decoder obtains the current prediction block based on the motion vector of the current inter-frame coded block; And through the decoder, a filtered prediction block is obtained based on the filter and the current prediction block, wherein the plurality of reconstructed samples are located in the upper or left region adjacent to the current inter-frame coding block.

15. The method of claim 14, further comprising: In response to receiving a first flag indicating whether the plurality of reconstructed samples are in the upper region or the left region of the current inter-frame coding block, the decoder determines the plurality of reconstructed samples based on the first flag.

16. The method of claim 15, further comprising: Before receiving the first flag, the decoder receives a second flag indicating that the plurality of reconstructed samples are in one of the upper region or the left region of the current inter-frame coding block.

17. The method according to claim 14, wherein, The current inter-frame coding block will be decoded based on the Advanced Motion Vector Prediction (AMVP) mode.

18. The method according to claim 17, wherein, The step of obtaining the current prediction block based on the motion vector of the current inter-frame coded block through the decoder includes: obtaining the current prediction block from reference list L0 or L1 based on the motion vector of the current inter-frame coded block.

19. The method of claim 14, wherein, The filter coefficients of the filter include: scaling factor and offset.

20. A method for video decoding, comprising: The decoder determines adjacent or non-adjacent neighboring blocks of the current inter-frame coding block, wherein the adjacent or non-adjacent neighboring blocks include multiple reconstructed samples that are adjacent or non-adjacent to the current inter-frame coding block; the decoder obtains multiple predicted samples of some of the multiple reconstructed samples; the decoder obtains a filter based on the multiple predicted samples and the partial reconstructed samples; the decoder obtains a current prediction block based on the motion vector of the current inter-frame coding block; and the decoder obtains a filtered prediction block based on the filter and the current prediction block.

21. The method according to claim 20, wherein, The partial reconstruction samples among the plurality of reconstruction samples include: reconstruction samples in the bottom boundary of the adjacent or non-adjacent neighboring blocks; or reconstruction samples in the right boundary of the adjacent or non-adjacent neighboring blocks; or reconstruction samples at the sample location selected based on the sub-sampling scheme in the adjacent or non-adjacent neighboring blocks.

22. The method according to claim 21, wherein, The sample point locations selected based on the subsampling scheme include: every Nth sample point in the horizontal direction and every Mth sample point in the vertical direction, where N and M are integers ≥ 2.

23. The method of claim 21, further comprising: The decoder receives a first syntax element, which indicates whether a portion of the reconstructed samples among the plurality of reconstructed samples are used to derive the filter. Alternatively, the decoder may receive a second syntax element, which indicates the sample location of a portion of the reconstructed samples from the plurality of reconstructed samples.

24. A method for video decoding, comprising: The decoder determines adjacent or non-adjacent neighboring blocks of the current inter-frame coding block, wherein the adjacent or non-adjacent neighboring blocks include multiple reconstructed samples that are adjacent or non-adjacent to the current inter-frame coding block; the decoder obtains multiple predicted samples of the multiple reconstructed samples; the decoder obtains a filter based on the multiple predicted samples and the multiple reconstructed samples, wherein the coefficients of the filter include an offset; the decoder calculates an offset adjustment for the offset based on the coefficients of the filter, the multiple predicted samples, the multiple reconstructed samples, and the total number of samples in the adjacent or non-adjacent neighboring blocks; the decoder adjusts the coefficients of the filter based on the offset adjustment; the decoder obtains a current prediction block based on the motion vector of the current inter-frame coding block; and the decoder obtains a filtered prediction block based on the filter adjusted according to the offset adjustment and the current prediction block.

25. The method of claim 24, further comprising: The decoder receives a flag indicating that the offset of the filter needs to be adjusted.

26. A method for video encoding, comprising: The encoder determines non-adjacent neighbor blocks of the current inter-frame coding block, wherein the non-adjacent neighbor blocks include multiple reconstructed samples that are not adjacent to the current inter-frame coding block; the encoder obtains multiple predicted samples of the non-adjacent neighbor blocks based on the motion vectors of the non-adjacent neighbor blocks; the encoder obtains a filter based on the multiple predicted samples and the multiple reconstructed samples; the encoder obtains a current prediction block based on the motion vectors of the current inter-frame coding block; and the encoder obtains a filtered prediction block based on the filter and the current prediction block.

27. The method according to claim 26, wherein, The step of obtaining multiple predicted samples of the non-adjacent neighboring blocks based on the motion vectors of the non-adjacent neighboring blocks includes: obtaining the multiple predicted samples of the non-adjacent neighboring blocks from a reference list L0 or L1.

28. The method according to claim 26, wherein, The step of obtaining multiple predicted samples of the non-adjacent neighboring blocks based on their motion vectors includes: obtaining multiple first predicted samples of the non-adjacent neighboring blocks from reference list L0; and obtaining multiple second predicted samples of the non-adjacent neighboring blocks from reference list L1.

29. The method according to claim 28, wherein, The step of obtaining a filter based on the plurality of predicted samples and the plurality of reconstructed samples includes: obtaining the plurality of predicted samples of the non-adjacent neighboring blocks based on the first predicted samples and the second predicted samples of the plurality of reconstructed samples; and obtaining a filter by minimizing the difference between the plurality of predicted samples of the non-adjacent neighboring blocks and the plurality of reconstructed samples, wherein the step of obtaining a current prediction block based on the motion vector of the current inter-frame coding block includes: obtaining a first current prediction block from reference list L0 based on the motion vector of the current inter-frame coding block; obtaining a second current prediction block from reference list L1 based on the motion vector of the current inter-frame coding block; and obtaining the current prediction block based on the first current prediction block and the second current prediction block, wherein the step of obtaining a filtered prediction block based on the filter and the current prediction block includes: obtaining the filtered prediction block by applying the filter to the current prediction block.

30. The method according to claim 28, wherein, The step of obtaining a filter based on the plurality of predicted samples and the plurality of reconstructed samples includes: obtaining a first filter by minimizing the difference between the plurality of first predicted samples and the plurality of reconstructed samples; and obtaining a second filter by minimizing the difference between the plurality of second predicted samples and the plurality of reconstructed samples; wherein, the step of obtaining a filtered prediction block based on the filter and the current prediction block includes: obtaining a first current prediction block and a second current prediction block from reference list L1 and list L0 respectively based on the motion vector of the current inter-frame coding block; obtaining a first filtered prediction block by applying the first filter to the first current prediction block; obtaining a second filtered prediction block by applying the second filter to the second current prediction block; and obtaining the filtered prediction block based on the first filtered prediction block and the second filtered prediction block.

31. The method according to claim 26, wherein, The filter is obtained based on the multiple predicted samples and the multiple reconstructed samples: multiple target reconstructed samples are calculated based on the multiple reconstructed samples and the previous filtered predictions of the multiple reconstructed samples; the coefficients of the current filter are obtained by minimizing the difference between the current prediction of the multiple reconstructed samples and the multiple target reconstructed samples. And by applying the current filter to the current prediction, the filtered prediction of the plurality of reconstructed samples is calculated.

32. The method of claim 26, further comprising: The encoder sends a first index via a signal to instruct the decoder to determine the non-adjacent neighboring blocks in the candidate list based on the first index.

33. The method according to claim 32, wherein, The candidate list is reordered based on the distortion of multiple neighboring reconstructed samples of the current inter-frame coding block calculated using motion information and filters associated with each non-adjacent neighboring block candidate.

34. The method according to claim 32, wherein, The candidate list is generated by merging non-neighboring block candidates associated with the corresponding adaptive filter coefficients into an existing merged candidate list.

35. The method according to claim 34, wherein, The existing merge candidate list includes the following candidate lists: Spatial Motion Vector Prediction MVP, Temporal Motion Vector Prediction TMVP, Non-adjacent MVP, History-based Motion Vector Prediction HMVP, Paired Merge Candidates, or Zero Candidates.

36. The method of claim 32, further comprising: The encoder sends a second index via a signal to instruct the decoder to use the candidate list to determine the non-adjacent neighboring blocks in the candidate list.

37. The method according to claim 32, wherein, Non-neighboring neighbor block candidates encoded using the sub-block-based temporal motion vector prediction SbTMVP mode or the geometric segmentation mode GPM mode are excluded from the candidate list.

38. A method for video encoding, comprising: The encoder obtains the filter coefficients of the filters applied to the previous coded blocks in the current inter-frame coded block. The encoder obtains a current prediction block based on the motion vector of the current inter-frame coding block; and the encoder obtains a filtered prediction block based on the filter and the current prediction block; wherein the filter is derived based on multiple prediction samples of non-adjacent neighboring blocks of the previous coding block and the non-adjacent neighboring blocks, the non-adjacent neighboring blocks including multiple reconstructed samples that are not adjacent to the previous coding block.

39. A method for video encoding, comprising: The encoder obtains a filter based on multiple reconstructed samples of the current inter-frame coding block and multiple predicted samples of the multiple reconstructed samples; The encoder obtains a current prediction block based on the motion vector of the current inter-frame coding block; and the encoder obtains a filtered prediction block based on the filter and the current prediction block, wherein the plurality of reconstructed samples are located in the upper or left region adjacent to the current inter-frame coding block.

40. The method of claim 39, further comprising: The encoder sends a first flag via a signal to indicate whether the plurality of reconstructed samples are in the upper region or the left region of the current inter-frame coding block.

41. The method of claim 40, further comprising: The encoder sends a second flag via a signal, the second flag indicating that the plurality of reconstructed samples are in one of the upper region or the left region of the current inter-frame coding block.

42. The method according to claim 39, wherein, The current inter-frame coding block will be encoded according to the Advanced Motion Vector Prediction (AMVP) mode.

43. The method according to claim 42, wherein, The step of obtaining the current prediction block based on the motion vector of the current inter-frame coding block through the encoder includes: obtaining the current prediction block from reference list L0 or L1 based on the motion vector of the current inter-frame coding block.

44. The method according to claim 39, wherein, The filter coefficients of the filter include: scaling factor and offset.

45. A method for video encoding, comprising: The encoder determines adjacent or non-adjacent neighbor blocks of the current inter-frame coding block, wherein the adjacent or non-adjacent neighbor blocks include multiple reconstructed samples that are adjacent or non-adjacent to the current inter-frame coding block; the encoder obtains multiple predicted samples of some of the multiple reconstructed samples; the encoder obtains a filter based on the multiple predicted samples and the partial reconstructed samples; the encoder obtains a current prediction block based on the motion vector of the current inter-frame coding block; and the encoder obtains a filtered prediction block based on the filter and the current prediction block.

46. ​​The method according to claim 45, wherein, The partial reconstruction samples among the plurality of reconstruction samples include: reconstruction samples in the bottom boundary of the adjacent or non-adjacent neighboring blocks; or reconstruction samples in the right boundary of the adjacent or non-adjacent neighboring blocks; or reconstruction samples at the sample location selected based on the sub-sampling scheme in the adjacent or non-adjacent neighboring blocks.

47. The method according to claim 46, wherein, The sample point locations selected based on the subsampling scheme include: every Nth sample point in the horizontal direction and every Mth sample point in the vertical direction, where N and M are integers ≥ 2.

48. The method of claim 46, further comprising: The encoder sends a first syntax element via a signal, the first syntax element indicating whether a portion of the plurality of reconstructed samples are used to derive the filter; Alternatively, the encoder may be used to send a second syntax element, which indicates the sample location of the portion of the reconstructed samples from the plurality of reconstructed samples.

49. A method for video encoding, comprising: The encoder determines adjacent or non-adjacent neighbor blocks of the current inter-frame coding block, wherein the adjacent or non-adjacent neighbor blocks include multiple reconstructed samples that are adjacent or non-adjacent to the current inter-frame coding block; the encoder obtains multiple predicted samples of the multiple reconstructed samples; the encoder obtains a filter based on the multiple predicted samples and the multiple reconstructed samples, the coefficients of the filter including an offset; the encoder calculates an offset adjustment for the offset based on the coefficients of the filter, the multiple predicted samples, the multiple reconstructed samples, and the total number of samples in the adjacent or non-adjacent neighbor blocks; the encoder adjusts the coefficients of the filter based on the offset adjustment; the encoder obtains a current prediction block based on the motion vector of the current inter-frame coding block; and the encoder obtains a filtered prediction block based on the filter adjusted according to the offset adjustment and the current prediction block.

50. The method of claim 49, further comprising: The encoder sends a signal indicating that the offset of the filter needs to be adjusted.

51. An apparatus for video decoding, comprising: One or more processors; and a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors, wherein the one or more processors are configured to perform the method according to any one of claims 1-25 when executing the instructions.

52. A non-transitory computer-readable storage medium for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to perform the method according to any one of claims 1-25.

53. An apparatus for video encoding, comprising: One or more processors; and a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors, wherein the one or more processors are configured to perform the method according to any one of claims 26-50 when executing the instructions.

54. A non-transitory computer-readable storage medium for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to perform the method according to any one of claims 26-50.

55. A non-transitory computer-readable storage medium for storing a bit stream to be decoded by the method according to any one of claims 1-25.

56. A non-transitory computer-readable storage medium for storing a bit stream generated by the method according to any one of claims 26-50.

57. A method for storing a bit stream, wherein, The bitstream includes encoded video information that is decoded by the method according to any one of claims 1-25 or encoded video information generated by the method according to any one of claims 26-50.

58. A method for transmitting a bit stream, wherein, The bitstream includes encoded video information that is decoded by the method according to any one of claims 1-25 or encoded video information generated by the method according to any one of claims 26-50.

Images