Use of local illumination compensation and delta motion vector refinement in video coding

Abstract

Example apparatuses, methods, and computer-readable media are described herein. An example method includes determining to decode a current block of video data using a merge mode. The example method includes determining to apply local illumination compensation (LIC) for a current block. The example method includes determining to apply decoder-side motion vector refinement (DMVR) for a current block. The example method includes decoding the current block based on the application of the LIC and the application of the DMVR.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 882,291, filed September 11, 2024; U.S. Provisional Patent Application No. 63 / 588,557, filed October 6, 2023; and U.S. Provisional Patent Application No. 63 / 605,138, filed December 1, 2023, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 18 / 882,291, filed September 11, 2024, claims the benefit of U.S. Provisional Patent Application No. 63 / 588,557, filed October 6, 2023; and U.S. Provisional Patent Application No. 63 / 605,138, filed December 1, 2023. Technical Field

[0002] This disclosure relates to video encoding and video decoding. Background Technology

[0003] Digital video capabilities can be incorporated into a wide variety of devices, including digital televisions, digital live broadcast systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones (so-called "smartphones"), video conferencing equipment, video streaming devices, and more. Digital video devices implement video decoding technologies, such as those defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 (Part 10, Advanced Video Decoding (AVC)), ITU-T H.265 / High Efficiency Video Decoding (HEVC), ITU-T H.266 / Variety Video Decoding (VVC) and extensions to these standards, as well as proprietary video codecs / formats such as AOMedia Video1 (AV1) developed by the Open Media Alliance. By implementing such video decoding technologies, video devices can more efficiently send, receive, encode, decode, and / or store digital video information.

[0004] Video decoding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video decoding, video slices (e.g., video pictures or portions of video pictures) can be divided into video blocks, which may also be referred to as decoding tree units (CTUs), decoding units (CUs), and / or decoding nodes. Video blocks in a slice after intra-frame decoding (I) of a picture are encoded using spatial prediction relative to reference samples in adjacent blocks within the same picture. Video blocks in a slice after inter-frame decoding (P or B) of a picture can use spatial prediction relative to reference samples in adjacent blocks within the same picture or temporal prediction relative to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame. Summary of the Invention

[0005] Generally, this disclosure describes techniques for inter-frame prediction in video codecs. Specifically, this disclosure describes techniques relating to the interaction between Local Illumination Compensation (LIC) and merging modes. Currently, in some video decoding standards and video codecs, for merging modes, the LIC flag and / or the Bidirectional Prediction (BCW) index with CU-level weights are always inherited from neighboring blocks. However, this may not be the optimal solution, as illumination changes may occur in neighboring blocks but not in the current block, or vice versa. The techniques of this disclosure address this problem by not requiring LIC information to be inherited from neighboring blocks, thereby potentially improving decoding quality, performance, and / or the like.

[0006] In one example, a method includes: determining a current block of video data to be decoded using a merge mode; obtaining a flag from the bitstream; determining a second merge list to use for the current block based on the value of the flag, wherein the second merge list is based on a first merge list of the two merge lists; and decoding the current block based on the second merge list.

[0007] In another example, a device includes one or more processors coupled to one or more memories, the one or more processors being configured to: determine a current block of video data to be decoded using a merging mode; obtain a flag from a bitstream; determine a second merging list to be used for the current block based on the value of the flag, wherein the second merging list is based on a first merging list of the two merging lists; and decode the current block based on the second merging list.

[0008] In another example, a non-transitory computer-readable medium storage instruction, when executed, causes one or more processors to: determine to decode a current block of video data using a merging mode; obtain a flag from the bitstream; determine, based on the value of the flag, to use a second merging list from two merging lists for the current block, wherein the second merging list is based on a first merging list from the two merging lists; and decode the current block based on the second merging list.

[0009] In one example, a method includes: determining a current block of video data to be decoded using a merge mode; generating a first merge list for the current block, wherein generating the first merge list includes applying template matching to candidates in the first merge list; generating a second merge list based on the first merge list; and decoding the current block using the merge mode and based on either the first or the second merge list.

[0010] In another example, a device includes one or more processors coupled to one or more memories, the one or more processors being configured to: determine a current block of video data to be decoded using a merging mode; generate a first merging list for the current block including applying template matching to candidates in the first merging list; generate a second merging list based on the first merging list; and decode the current block using the merging mode and based on either the first merging list or the second merging list.

[0011] In another example, a non-transitory computer-readable medium storage instruction, when executed, causes one or more processors to: determine a current block of video data to be decoded using a merging mode; generate a first merging list for the current block including applying template matching to candidates in the first merging list; generate a second merging list based on the first merging list; and decode the current block using the merging mode and based on either the first or the second merging list.

[0012] In one example, a method includes: determining to decode the current block of video data using a merging mode; determining to apply Local Illumination Compensation (LIC) to the current block; determining to apply Decoder-Side Motion Vector Refinement (DMVR) to the current block; and decoding the current block based on the application of LIC and the application of DMVR.

[0013] In another example, a device includes one or more processors coupled to one or more memories, the one or more processors being configured to: determine to decode a current block of video data using a merging mode; determine to apply Local Illumination Compensation (LIC) to the current block; determine to apply Decoder-Side Motion Vector Refinement (DMVR) to the current block; and decode the current block based on the application of LIC and the application of DMVR.

[0014] In another example, a non-transitory computer-readable medium storage instruction, when executed, causes one or more processors to: determine to decode a current block of video data using a merge mode; determine to apply Local Illumination Compensation (LIC) to the current block; determine to apply Decoder-Side Motion Vector Refinement (DMVR) to the current block; and decode the current block based on the application of LIC and the application of DMVR.

[0015] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0016] Figure 1 This is a block diagram illustrating an example video encoding and decoding system that can perform the techniques of this disclosure.

[0017] Figure 2 This is a conceptual diagram illustrating example localization of candidate instances for merging instances in the instance space.

[0018] Figure 3 This is a conceptual diagram illustrating example candidate pairs for considering redundancy checks for spatial merging candidates.

[0019] Figure 4 This is a conceptual diagram illustrating an example of motion vector scaling for time merging candidates.

[0020] Figure 5 This is a conceptual diagram illustrating example candidate localization for time merging candidates.

[0021] Figure 6 This is a conceptual diagram illustrating an example of a spatially adjacent block used to derive spatial merging candidates.

[0022] Figure 7 This is a conceptual diagram illustrating an example MMVD search point.

[0023] Figure 8 This is a conceptual diagram illustrating an example of bilateral matching.

[0024] Figure 9 This is a conceptual diagram illustrating an example of template matching performed on the search area surrounding the initial MV.

[0025] Figure 10 This is a conceptual diagram illustrating an example sub-block template used to generate SbTMVP.

[0026] Figure 11 This is a conceptual diagram illustrating a regression-based decoder with positive and negative affine derivation techniques.

[0027] Figures 12A to 12C This is a conceptual diagram illustrating an example of the three stages of a non-translational affine parameter search.

[0028] Figure 13 This is a conceptual diagram illustrating a sample standalone search for CPMV.

[0029] Figure 14 This is a conceptual diagram illustrating an example diamond-shaped region within the search area.

[0030] Figure 15 This is a conceptual diagram illustrating an example extended CU region used in BDOF.

[0031] Figure 16 This is a flowchart illustrating an example signaling technique for using either a first merge list or a second merge list, according to one or more aspects of this disclosure.

[0032] Figure 17 This is a flowchart illustrating an example technique for template-matching-based local lighting compensation and a second merge list, according to one or more aspects of this disclosure.

[0033] Figure 18 This is a flowchart illustrating a technique for using local illumination compensation with differential motion vector refinement according to one or more aspects of this disclosure.

[0034] Figure 19 This is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure.

[0035] Figure 20 This is a block diagram illustrating an example video decoder that can perform the techniques of this disclosure.

[0036] Figure 21 This is a flowchart illustrating an example method for encoding the current block according to the technology of this disclosure.

[0037] Figure 22 This is a flowchart illustrating an example method for decoding the current block according to the technology of this disclosure. Detailed Implementation

[0038] Generally, this disclosure describes techniques for inter-frame prediction in video codecs. Specifically, this disclosure describes techniques relating to the interaction between Local Illumination Compensation (LIC) and merging modes. Currently, in some video decoding standards and video codecs, for merging modes, the LIC flag and / or BCW index are always inherited from neighboring blocks. However, this may not be the optimal solution, as illumination changes may occur in neighboring blocks but not in the current block, or vice versa. The techniques of this disclosure address this problem by not requiring LIC information to be inherited from neighboring blocks. For example, the techniques of this disclosure may include using a first merge list and a second merge list, instead of a single merge list. The second merge list may be based on the first merge list. Thus, the first and second merge lists together should not be confused with the L0 and L1 lists used in bidirectional prediction. The second merge list may have associated LIC flags, and the second merge list may include LIC flags with values ​​different from one or more values ​​of the LIC flags associated with the first merge list. The techniques of this disclosure can improve decoding quality, performance, and / or the like. The merge list may also be alternatively referred to herein as a “candidate list” or a “merging candidate list.”

[0039] Figure 1 This is a block diagram illustrating an example video encoding and decoding system 100 capable of performing the techniques of this disclosure. The techniques of this disclosure generally involve decoding (encoding and / or decoding) video data. Typically, video data includes any data used for processing video. Therefore, video data may include unencoded raw video, encoded video, decoded (e.g., reconstructed) video, and video metadata, such as signaling data.

[0040] like Figure 1 As shown, in this example, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. Specifically, source device 102 provides video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can be or may include any of a wide range of devices, such as desktop computers, laptop computers, mobile devices, tablet computers, set-top boxes, mobile phones (such as smartphones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and destination device 116 may be configured for wireless communication and are therefore referred to as wireless communication devices.

[0041] exist Figure 1In the example, source device 102 includes a video source 104, memory 106, video encoder 200, and output interface 108. Destination device 116 includes an input interface 122, video decoder 300, memory 120, and display device 118. According to this disclosure, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to be used with a technology that interacts between LIC and merge modes. Thus, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, the source device and destination device may include other components or arrangements. For example, source device 102 may receive video data from an external video source such as an external camera. Similarly, destination device 116 may interface with an external display device, rather than including an integrated display device.

[0042] like Figure 1 The system 100 shown is merely an example. Typically, any digital video encoding and / or decoding device can perform techniques for interaction between LIC and merge modes. Source device 102 and destination device 116 are merely examples of such decoding devices, where source device 102 generates decoded video data for transmission to destination device 116. This disclosure refers to a “decoding” device as a device that performs the decoding (e.g., encoding and / or decoding) of data. Thus, video encoder 200 and video decoder 300 represent examples of decoding devices, specifically, a video encoder and a video decoder, respectively. In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes video encoding and decoding components. Therefore, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0043] Typically, video source 104 represents a source of video data (e.g., unencoded raw video data) and provides a sequential series of pictures (also referred to as “frames”) of the video data to video encoder 200, which encodes the data of the pictures. Video source 104 of source device 102 may include video capture devices such as cameras, video archives containing previously captured raw video, and / or video feed interfaces for receiving video from video content providers. Alternatively, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes the captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from the received order (sometimes referred to as “display order”) to a decoding order for decoding. Video encoder 200 may generate a bitstream comprising the encoded video data. Then, the source device 102 can output the encoded video data to the computer-readable medium 110 via the output interface 108 for reception and / or retrieval by, for example, the input interface 122 of the destination device 116.

[0044] The memory 106 of source device 102 and the memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 may store software instructions executable by, for example, video encoder 200 and video decoder 300. Although memories 106 and 120 are shown separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store encoded video data, such as output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106 and 120 may be allocated as one or more video buffers, for example, to store raw decoded and / or encoded video data.

[0045] Computer-readable medium 110 may represent any type of medium or device capable of transmitting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to directly transmit encoded video data to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. According to a communication standard such as a wireless communication protocol, output interface 108 may modulate the transmitted signal including the encoded video data, and input interface 122 may demodulate the received transmitted signal. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium may include a router, switch, base station, or any other equipment that may be useful for facilitating communication from source device 102 to destination device 116.

[0046] In some examples, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any data storage medium of various distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0047] In some examples, source device 102 can output encoded video data to file server 114 or another intermediate storage device that can store the encoded video data generated by source device 102. Destination device 116 can access the stored video data from file server 114 via streaming or download.

[0048] File server 114 can be any type of server device capable of storing encoded video data and sending the encoded video data to destination device 116. File server 114 may represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or FLUTE-based file delivery protocol), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a Network Attached Storage (NAS) device. File server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as HTTP-based Dynamic Adaptive Streaming (DASH), HTTP Live Streaming (HLS), Real-Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc.

[0049] Destination device 116 can access encoded video data from file server 114 via any standard data connection, including an internet connection. This may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on file server 114. Input interface 122 can be configured to operate according to any or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0050] Output interface 108 and input interface 122 can represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 can be configured to transmit data such as encoded video data according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc. In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 can be configured according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee), etc. ™ ),Bluetooth ™Standards are used to transmit data such as encoded video data. In some examples, source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functions belonging to video encoder 200 and / or output interface 108, and destination device 116 may include an SoC device for performing functions belonging to video decoder 300 and / or input interface 122.

[0051] The technology disclosed herein can be applied to video decoding to support any multimedia application in a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission (such as HTTP-based Dynamic Adaptive Streaming (DASH)), digital video encoded onto data storage media, decoding of digital video stored on data storage media, or other applications.

[0052] The input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200 and also used by the video decoder 300, such as syntax elements having values ​​describing the characteristics and / or processing of video blocks or other decoded units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays a decoded picture of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0053] Despite Figure 1Not shown, but in some examples, both the video encoder 200 and the video decoder 300 may be integrated with the audio encoder and / or audio decoder (e.g., audio codec), and may include appropriate MUX-DEMUX units or other hardware and / or software to process multiplexed streams that include both audio and video in a common data stream. Example audio codecs may include AAC, AC-3, AC-4, ALAC, ALS, AMBE, AMR, AMR-WB (G.722.2), AMR-WB+, aptX (various versions), ATRAC, BroadVoice (BV16, BV32), CELT, Enhanced AC-3 (E-AC-3), EVS, FLAC, G.711, G.722, G.722.1, G.722.2 (AMR-WB), G.723.1, G.726, G.728, G.729, G.729.1, GSM-FR, HE-AAC, iLBC, iSAC, LA Lyra, Monkey's Audio, MP1, MP2 (MPEG-1, 2 Audio Layer II), MP3, Musepack, Nellymoser Asao, OptimFROG, Opus, Sac, Satin, SBC, SILK, Siren 7, Speex, SVOPC, True Audio (TTA), TwinVQ, USAC, Vorbis (Ogg), WavPack and Windows Media Aud.

[0054] Both the video encoder 200 and the video decoder 300 can be implemented as any of a variety of suitable encoder and / or decoder circuits comprising a processing system, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic components, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of this disclosure. Each of the video encoder 200 and the video decoder 300 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder / decoder (CODEC) in the respective device. Devices including the video encoder 200 and / or the video decoder 300 may implement the video encoder 200 and / or the video decoder 300 in processing circuitry such as integrated circuits and / or microprocessors. Such devices may be wireless communication devices (such as cellular phones) or any other type of device described herein.

[0055] The video encoder 200 and video decoder 300 may operate according to a video decoding standard such as ITU-T H.265 (also known as High Efficiency Video Decoding (HEVC)) or extensions thereof such as Multi-View and / or Scalable Video Decoding Extensions. Alternatively, the video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards such as ITU-T H.266 (also known as Multi-Functional Video Decoding (VVC)). In other examples, the video encoder 200 and video decoder 300 may operate according to proprietary video codecs / formats such as AOMedia Video 1 (AV1), extensions to AV1, and / or subsequent versions of AV1 (e.g., AV2)). In other examples, the video encoder 200 and video decoder 300 may operate according to other proprietary formats or industry standards. However, the techniques disclosed herein are not limited to any particular decoding standard or format. Generally, the video encoder 200 and video decoder 300 may be configured to perform the techniques of this disclosure in conjunction with any video decoding technology using LIC and merging modes.

[0056] Typically, video encoder 200 and video decoder 300 perform block-based decoding of images. The term "block" generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used during encoding and / or decoding). For example, a block may include a two-dimensional matrix of samples of luminance and / or chrominance data. Typically, video encoder 200 and video decoder 300 decode video data represented in YUV (e.g., Y, Cb, Cr) format. That is, instead of decoding the red, green, and blue (RGB) data used for images, video encoder 200 and video decoder 300 decode the luminance and chrominance components, where the chrominance components may include both red hue chrominance components and blue hue chrominance components. In some examples, video encoder 200 converts the received RGB format data to a YUV representation before encoding, and video decoder 300 converts the YUV representation to RGB format. Alternatively, preprocessing and postprocessing units (not shown) may perform these conversions.

[0057] This disclosure generally relates to the decoding (e.g., encoding and decoding) of images to include processes of encoding or decoding data of the image. Similarly, this disclosure may relate to the decoding of blocks of images to include processes of encoding or decoding data used for the blocks (e.g., prediction and / or residual decoding). Encoded video bitstreams typically include a series of values ​​for syntax elements representing decoding decisions (e.g., decoding modes) and the partitioning of images into blocks. Therefore, references to the decoding of images or blocks should generally be understood as the decoded values ​​of the syntax elements that form the images or blocks.

[0058] HEVC defines various blocks, including decoding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video decoder (such as a video encoder 200) divides the decoding tree unit (CTU) into CUs based on a quadtree structure. That is, the video decoder divides the CTU and CU into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. Nodes without child nodes are called "leaf nodes," and the CU of such leaf nodes may include one or more PUs and / or one or more TUs. The video decoder may further divide the PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents the partitioning of the TU. In HEVC, the PU represents inter-frame prediction data, while the TU represents residual data. The CU after intra-frame prediction includes intra-frame prediction information, such as intra-frame mode indication.

[0059] As another example, video encoder 200 and video decoder 300 can be configured to operate according to VVC. According to VVC, the video decoder (such as video encoder 200) partitions the image into multiple CTUs. Video encoder 200 can partition the CTUs according to a tree structure (such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure removes the concept of multiple partition types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level partitioned according to a quadtree and a second level partitioned according to a binary tree. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to CUs.

[0060] In the MTT partitioning structure, blocks can be divided using quadtree (QT) partitioning, binary tree (BT) partitioning, and one or more types of ternary tree (TT) partitioning (also known as triplet tree (TT)). A ternary tree or triplet tree partition is a partition in which a block is divided into three sub-blocks. In some examples, a ternary tree or triplet tree partition divides a block into three sub-blocks without dividing the original block through the center. Partition types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0061] When operating according to the AV1 codec, the video encoder 200 and video decoder 300 can be configured to decode video data in blocks. In AV1, the largest decoded block that can be processed is called a superblock. In AV1, a superblock can be 128×128 luminance samples or 64×64 luminance samples. However, in subsequent video decoding formats (e.g., AV2), superblocks can be defined by different (e.g., larger) luminance sample sizes. In some examples, the superblock is the top level of a block quadtree. The video encoder 200 can further divide the superblock into smaller decoded blocks. The video encoder 200 can use square or non-square partitions to divide the superblock and other decoded blocks into smaller blocks. Non-square blocks can include N / 2×N blocks, N×N / 2 blocks, N / 4×N blocks, and N×N / 4 blocks. The video encoder 200 and video decoder 300 can perform separate prediction and transform processing for each decoded block.

[0062] AV1 also defines video data tiles. A tile is a rectangular array of superblocks that can be decoded independently of other tiles. That is, the video encoder 200 and video decoder 300 can encode and decode the decoding blocks within a tile separately without using video data from other tiles. However, the video encoder 200 and video decoder 300 can perform filtering across tile boundaries. The tile size can be uniform or non-uniform. Tile-based decoding enables parallel processing and / or multithreading in the encoder and decoder implementations.

[0063] In some examples, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luma component and another QTBT / MTT structure for the two chroma components (or two QTBT / MTT structures for the respective chroma components).

[0064] The video encoder 200 and the video decoder 300 can be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning or other partitioning structures.

[0065] In some examples, a CTU includes a decoded tree block (CTB) of luminance samples, two corresponding CTBs of chrominance samples of an image with three sample arrays, or a CTB of samples of an image decoded using three separate color planes and a syntax structure for decoding the samples. A CTB can be an N×N block of samples of some value N, such that a partitioning method divides the components into CTBs. A component is an array or a single sample from one of the three arrays (luminance and two chrominance) constituting a 4:2:0, 4:2:2, or 4:4:4 color format image, or an array or a single sample constituting an array or array constituting a monochrome format image. In some examples, a decoded block is an M×N block of samples of values ​​M and N, such that a partitioning method divides the CTB into decoded blocks.

[0066] Blocks (e.g., CTUs or CUs) can be grouped in various ways within an image. As an example, a brick can refer to a rectangular area of ​​a row of CTUs within a specific tile in an image. A tile can be a rectangular area of ​​CTUs within a specific tile column and a specific tile row in an image. A tile column refers to a rectangular area of ​​a CTU having a height equal to the height of the image and a width specified by syntax elements (e.g., such as in an image parameter set). A tile row refers to a rectangular area of ​​a CTU having a height specified by syntax elements (e.g., such as in an image parameter set) and a width equal to the width of the image.

[0067] In some examples, a tile can be divided into multiple bricks, each brick comprising one or more CTU rows within the tile. A tile that is not divided into multiple bricks can also be called a brick. However, bricks that are a true subset of a tile cannot be called a tile. Bricks in an image can also be arranged in slices. A slice can be an integer number of bricks in an image that can be uniquely contained within a single Network Abstraction Layer (NAL) unit. In some examples, a slice comprises multiple complete tiles or a consecutive sequence of complete bricks comprising only one tile.

[0068] This disclosure uses "N×N" and "N by N" interchangeably to refer to the sample size of a block (such as a CU or other video block) in the vertical and horizontal dimensions, for example, 16×16 samples or 16 by 16 samples. Generally, a 16×16 CU will have 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Similarly, an N×N CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily need to have the same number of samples in the horizontal direction as it does in the vertical direction. For example, a CU may include N×M samples, where M is not necessarily equal to N.

[0069] The video encoder 200 encodes video data representing prediction and / or residual information, as well as other information, for use in the control unit (CU). The prediction information indicates how the CU should be predicted to form a prediction block for the CU. The residual information typically represents the sample-by-sample difference between a sample of the CU before encoding and the prediction block.

[0070] To predict the Cubic Frame (CU), the video encoder 200 typically forms prediction blocks for the CU through inter-frame prediction or intra-frame prediction. Inter-frame prediction typically refers to predicting the CU from data in a previously decoded image, while intra-frame prediction typically refers to predicting the CU from data in a previously decoded image within the same frame. To perform inter-frame prediction, the video encoder 200 can use one or more motion vectors to generate prediction blocks. The video encoder 200 can typically perform a motion search to identify reference blocks that closely match the CU, for example, based on the differences between the CU and a reference block. The video encoder 200 can use sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to compute difference metrics to determine whether a reference block closely matches the current CU. In some examples, the video encoder 200 can use unidirectional or bidirectional prediction to predict the current CU.

[0071] Some examples of VVC also provide an affine motion compensation mode, which can be viewed as an inter-frame prediction mode. In affine motion compensation mode, the video encoder 200 can determine two or more motion vectors representing non-translational motion, such as zooming in or out, rotation, perspective motion, or other irregular motion types.

[0072] To perform intra-frame prediction, the video encoder 200 can select an intra-frame prediction mode to generate prediction blocks. Some examples of VVC provide sixty-seven intra-frame prediction modes, including various directional modes, as well as planar and DC modes. Typically, the video encoder 200 selects an intra-frame prediction mode that describes the neighboring samples of the current block (e.g., the block of the CU), from which it predicts samples for the current block. Assuming the video encoder 200 decodes the CTU and CU in raster scan order (from left to right, from top to bottom), such samples are typically located above, to the upper left, or to the left of the current block within the same frame as the current block.

[0073] The video encoder 200 encodes data representing the prediction mode of the current block. For example, for inter-frame prediction modes, the video encoder 200 may encode data indicating which of the various available inter-frame prediction modes is used, as well as the motion information for the corresponding mode. For example, for unidirectional or bidirectional inter-frame prediction, the video encoder 200 may encode motion vectors using Advanced Motion Vector Prediction (AMVP) or merging modes. The video encoder 200 may use similar modes to encode motion vectors used for affine motion compensation modes.

[0074] AV1 includes two common techniques for encoding and decoding blocks of video data. These two common techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when using intra-frame prediction modes to predict blocks of video data for the current frame, the video encoder 200 and video decoder 300 do not use video data from other frames of the video data. For most intra-frame prediction modes, the video encoder 200 encodes blocks of the current frame based on the difference between sample values ​​in the current block and predicted values ​​generated from reference samples in the same frame. The video encoder 200 determines the predicted values ​​generated from the reference samples based on the intra-frame prediction mode.

[0075] After prediction (such as intra-frame or inter-frame prediction for a block), the video encoder 200 can compute residual data for the block. The residual data (such as a residual block) represents the sample-by-sample difference between the block and the prediction block used to form the block, which is formed using the corresponding prediction mode. The video encoder 200 can apply one or more transforms to the residual block to produce transformed data in the transform domain rather than the sample domain. For example, the video encoder 200 can apply a Discrete Cosine Transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Additionally, the video encoder 200 can apply a secondary transform after the first transform, such as the Mode Correlated Inseparable Secondary Transform (MDNSST), the Signal Correlation Transform, the Karhunen-Loeve Transform (KLT), etc. The video encoder 200 produces transform coefficients after applying one or more transforms.

[0076] As noted above, after any transformation that produces the transform coefficients, the video encoder 200 may perform quantization on the transform coefficients. Quantization generally refers to a process in which the transform coefficients are quantized to reduce the amount of data used to represent them, thereby providing further compression. By performing the quantization process, the video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 may round an n-bit value down to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 may perform a bitwise right shift on the value to be quantized.

[0077] After quantization, the video encoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix containing the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) transform coefficients before the vector and lower-energy (and therefore higher-frequency) transform coefficients after the vector. In some examples, the video encoder 200 can utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector, and then entropy-encode the quantized transform coefficients of that vector. In other examples, the video encoder 200 can perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 can entropy-encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic decoding (CABAC). The video encoder 200 can also entropy-encode the values ​​of syntax elements describing metadata associated with the encoded video data, which is used by the video decoder 300 when decoding the video data.

[0078] To perform CABAC, the video encoder 200 can assign context within a context model to the symbols to be transmitted. Context may involve, for example, whether the neighboring values ​​of a symbol are zero. Probability determination can be based on the context assigned to the symbols.

[0079] The video encoder 200 may further generate syntax data for the video decoder 300, such as block-based syntax data, image-based syntax data, and sequence-based syntax data, for example, in image headers, block headers, and slice headers, or generate other syntax data such as sequence parameter sets (SPS), image parameter sets (PPS), or video parameter sets (VPS). The video decoder 300 may also decode such syntax data to determine how to decode the corresponding video data.

[0080] In this way, the video encoder 200 can generate a bitstream that includes encoded video data, such as syntax elements describing the partitioning of images into blocks (e.g., CUs) and prediction and / or residual information for the blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0081] Typically, the video decoder 300 performs the reverse process of the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 can use CABAC to decode the values ​​of syntax elements used for the bitstream in a manner substantially similar to, but reversed, the CABAC encoding process of the video encoder 200. Syntax elements can define partitioning information for dividing a picture into CTUs and for partitioning each CTU according to a corresponding partitioning structure such as a QTBT structure to define the CUs of the CTU. Syntax elements can further define prediction and residual information for video data blocks (e.g., CUs).

[0082] The residual information can be represented, for example, by quantized transform coefficients. The video decoder 300 can inversely quantize and inverse transform the quantized transform coefficients of the block to reconstruct the residual block for the block. The video decoder 300 uses a signaling prediction mode (intra-frame prediction or inter-frame prediction) and associated prediction information (e.g., motion information for inter-frame prediction) to form a prediction block for the block. The video decoder 300 can then combine the prediction block and the residual block (on a sample-by-sample basis) to reconstruct the original block. The video decoder 300 can perform additional processing, such as performing a deblocking process to reduce visual artifacts along the block boundaries.

[0083] This disclosure may generally relate to "signaling" certain information (such as syntax elements). The term "signaling" can generally refer to the communication of values ​​and / or other data of syntax elements used to decode encoded video data. That is, video encoder 200 may signal the values ​​of syntax elements in the bitstream. Generally, signaling refers to generating values ​​in the bitstream. As noted above, source device 102 may transmit the bitstream to destination device 116 substantially in real time or not in real time (such as when syntax elements are stored in storage device 112 for later retrieval by destination device 116).

[0084] According to the technology disclosed herein, a method includes: determining whether to apply a merging mode to a current block of video data; determining whether to apply LIC, independent of whether local illumination compensation (LIC) is applied to adjacent blocks; and decoding the current block based on the merging mode and the determination of whether LIC is applied.

[0085] Inter-frame prediction (LIC) is a technique that models the local illumination variation between the current block and its predicted block as a function of the current block template and the reference block template. The parameters of the function can be represented by the scale α and the offset β, forming a linear equation, i.e., α... p[x]+β is used to compensate for illumination variations, where p[x] is the reference sample pointed to by the MV at position x on the reference image. Since α and β can be derived based on the current block template and the reference block template, they require no signaling overhead except for signaling the LIC flag for AMVP mode to indicate the use of LIC.

[0086] JVET-O0066 (V. Seregin, W.-J. Chien, T. Hsieh, N. Hu, M. Karczewicz, C.-M. Tsai, C.-C. Chen, C.-W. Hsu, Y.-W. Huang, S.-M. Lei, H. Chen, X. Ma, H. Yang, “CE4-3.1a and CE4-3.1b: Unidirectional local illumination compensation with affine prediction”, JVET-O0066, An example of local illumination compensation proposed in June 2019 can be used for unidirectional prediction of inter-frame CUs with the following modifications: intra-frame adjacent samples can be used in LIC parameter derivation; LIC is disabled for blocks with fewer than 32 luminance samples; for both non-sub-block mode and affine mode, LIC parameter derivation is performed based on template block samples corresponding to the current CU instead of partial template block samples corresponding to the first top-left 16x16 cell; and the samples of the reference block template are generated using MC and block MV without rounding to integer pixel precision.

[0087] In JVET-AD0213 (“EE2-Test2.7: Improvements on local illumination compensation” by X. Xiu, N. Yan, C. Ma, H.-J. Jhu, C.-W. Kuo, W. Chen, and X. Wang, JVET-AD0213, April 2023), the LIC model is extended to bidirectional prediction of the CU and is adopted in the enhanced compression model (ECM), where two different linear models are applied to two prediction blocks, which are then combined to generate bidirectional prediction samples of the current CU, for example...

[0088]

[0089] and

[0090]

[0091]

[0092] in, and as well as and Indicates the scale and offset in L0 and L1 respectively; The weights of the weighted combination of L0 and L1 predictions (as indicated by the CU-level BCW index).

[0093] The LIC technique with bidirectional prediction CU first minimizes the prediction between L0 templates. With template The difference between them is used to derive the L0 parameter, and by subtracting Update the corresponding samples in The samples in the dataset. Then, the minimum value between L1 template prediction and the target value is calculated. The L1 parameter represents the difference between the current template and the updated template. Finally, the L0 parameter is refined again in the same way.

[0094] According to the current LIC design, for AMVP bidirectional prediction CUs, a flag is signaled to indicate LIC mode, while for merge-related inter-frame CUs, this flag is inherited. Additionally, LIC is disabled for decoder-side motion vector refinement (DMVR) and BDOF.

[0095] In one example of ECM, the derived LIC model parameters are stored in the CU because, when OBMC is executed, the LIC model parameters can be additionally compared to determine whether OBMC should be executed. Therefore, for a CU where the LIC flag is equal to "true" (e.g., 1), a set of LIC model parameters is stored and available for future use.

[0096] An example of Non-EE2: Enhancements on Local Illumination Compensation (NLIC) is proposed in X. Xiu, C. Ma, N. Yan, H.-J. Jhu, C.-W. Kuo, W. Chen, and X. Wang (Hanover, Germany, October 13-20, 2023, JVET-AF0191). For this technique, samples of previously decoded CUs are used to derive a linear model for motion compensation of the current block, instead of using template samples. Specifically, after reconstructing each inter-frame CU (except for Geometric Partition Mode (GPM) and Sub-Block Temporal Motion Vector Prediction (SbTMVP) CUs), a linear model is derived by minimizing the difference between the reconstructed and predicted samples of the block. The derived LIC model parameters are also stored. This LIC model is derived independently of the block's LIC flag value. If the CU's LIC flag is "true," two sets of LIC model parameters are stored. One set of LIC model parameters is derived between the current template and the reference template. Another set of LIC model parameters is derived by minimizing the difference between the reconstructed samples and the predicted samples of the block. If the LIC flag is "pseudo" (e.g., 0), only one set of LIC parameters is stored, derived between the reconstructed samples and the predicted samples of the block.

[0097] The paper “EE2-3.2: LIC flagderivation for merge candidates with template costs” by N. Zhang, K. Zhang, H. Liu, Y. Wang, and L. Zhang (Hanover, Germany, October 13-20, 2023, JVET-AF0128) proposes a template-matching cost-based LIC flag derivation technique, which has been adopted by the ECM reference software. In this technique, the template-matching cost is calculated twice for the same merge candidate, setting the LIC flag to “true” or “false” each time. The two template-matching costs are compared, and a predefined threshold is used to determine whether to modify the LIC flag. Currently, this technique is only applicable to unidirectional prediction of merge candidates.

[0098] The standard merging mode is now described. The video encoder 200 or video decoder 300 can derive spatial merge candidates. The derivation of spatial merge candidates in VVC is the same as that in HEVC.

[0099] Figure 2This is a conceptual diagram illustrating example positioning of candidate instances for space merging. For the current block 126, either the video encoder 200 or the video decoder 300 can be located in... Figure 2 A maximum of four merging candidates are selected from the candidates shown for positioning. The order of derivation or selection is A0, B0, B1, A1, and B2. Positioning B2 is considered only if any CU for positioning A0, B0, B1, or A1 is unavailable (e.g., because it belongs to another slice or tile) or if intra-frame decoding is required. After the candidate for positioning A1 is added to the merging list, the addition of the remaining candidates undergoes a redundancy check, which ensures that candidates with the same motion information are excluded from the merging list, thereby improving decoding efficiency.

[0100] Figure 3 This is a conceptual diagram illustrating example candidate pairs considered for redundancy checks in spatial merging candidates. To reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy checks. Instead, only... Figure 3 The pairs connected by arrows 128 or 130 in the list are added only if the candidates used for redundancy checking do not have the same motion information. For example, if the corresponding candidates have the same motion information, the video encoder 200 or video decoder 300 may not add the corresponding candidate to the merge list.

[0101] Figure 4 This is a conceptual diagram illustrating an example of motion vector scaling for time merging candidates. In some examples, a video encoder 200 or a video decoder 300 can derive time candidates. In some examples, only one candidate is added to the merge candidate list. Specifically, in the derivation of time merging candidates, scaled motion vectors are derived based on co-located CUs belonging to co-located reference images. The list of reference images to be used for deriving co-located CUs is explicitly signaled in the slice header. Scaled motion vector 135 for the time merging candidate is obtained, as shown... Figure 4 The dashed lines illustrate this. The scaled motion vector 135 of the current CU 134 is scaled from the motion vector 137 of co-located CU 138 using Picture Order Count (POC) distances tb and td. The POC distance tb is defined as the POC difference between the current reference image 140 and the current image 132. The POC difference td is defined as the POC difference between the reference image 142 and the co-located image 136. The reference image index for the temporal merging candidate is set to zero.

[0102] Figure 5 This is a conceptual diagram illustrating example candidate positioning for time merging candidates. Positioning of the current CU 134 (e.g., which could be a luminance (Y) block) is selected between candidates C0 144 and C1 146, as shown below. Figure 5As depicted. If the CU at location C0 144 is unavailable, intra-frame decoded, or located outside the current CTU line, then location C1 146 is used. Otherwise, location C0 144 is used to derive time merging candidates.

[0103] The video encoder 200 or video decoder 300 can derive one or more history-based merging candidates. Motion information from previously decoded blocks is stored in a table (HMVP table) and used as the MVP of the current CU. The table, with multiple entries, is maintained during the encoding / decoding process. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to determine if a duplicate candidate exists in the table. If a duplicate candidate is found, it is removed from the table, and all other candidates following the duplicate candidate are moved forward in the table.

[0104] After merging the spatial and temporal candidates, HMVP candidates are added to the merge list (also referred to as the candidate list in this paper). The latest few HMVP candidates in the table are checked sequentially, and these latest HMVP candidates are inserted into the candidate list after the TMVP candidates. Some redundancy checks may be applied to the HMVP candidates before insertion into the candidate list.

[0105] The video encoder 200 or video decoder 300 can derive paired average merge candidates. Paired average merge candidates are generated by averaging predefined candidate pairs in an existing merge list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in a list, they are averaged even if they point to different reference images; if only one motion vector is available, that specific motion vector is used directly; if no motion vector is available, the list remains invalid.

[0106] When the merge list is not full after adding pairs of average merge candidates, insert zero MVPs at the end until the maximum number of merge candidates is reached.

[0107] The video encoder 200 or video decoder 300 can derive non-adjacent spatial candidates. In ECM, a new type of regular merging candidate, namely the non-adjacent spatial merging candidate, is introduced. Similar to JVET-L0399, in the regular merging candidate list, the non-adjacent spatial merging candidate is inserted after the Temporal Motion Vector Prediction (TMVP).

[0108] Figure 6 This is a conceptual diagram illustrating an example of a spatially adjacent block used to derive spatial merging candidates. Figure 6 The pattern for spatial merging candidates is shown in the diagram. The distance between a non-adjacent spatial candidate and the current decoded block is based on the width and height of the current decoded block. Line buffer constraints are not applied.

[0109] The merging pattern with Motion Vector Difference (MMVD) is now described. In addition to the merging pattern that directly uses implicitly derived motion information to generate prediction samples for the current CU (sometimes called the regular merging pattern), MMVD is introduced in VVC. In MMVD, after selecting merging candidates, the merging candidates are further refined by signaled motion vector difference (MVD) information. MVD information includes a merging candidate flag, an index specifying the motion magnitude value, and an index indicating the motion direction. In the MMVD pattern, one of the top two candidates in the merging list is selected as the basis MV. The merging candidate flag is signaled to specify which of the top two candidates to use.

[0110] Figure 7 This is a conceptual diagram illustrating an example MMVD search point. The distance index specifies motion value information and indicates a predefined offset from the starting point. For example... Figure 7 As shown, the offset is added to the horizontal or vertical component of the starting or base MV. Table 1-1 specifies the relationship between the distance index and the predefined offset.

[0111] Table 1-1

[0112]

[0113] The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent any of the four directions, as shown in Table 1-2. It should be noted that the meaning of the MVD sign can vary depending on the specific information of the starting MV. When the starting MV is a unidirectional or bidirectional predictive MV and both lists point to the same side of the current image (e.g., both reference POCs are greater than or less than the current image's POC), the sign in Table 1-2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional predictive MV and the two MVs point to different sides of the current image (e.g., one reference POC is greater than the current image's POC, and the other reference POC is less than the current image's POC), the sign in "Error! Reference source not found." specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of the list 1 MV has the opposite value.

[0114] Table 1-2

[0115]

[0116] In ECM, the number of directions is further increased from 4 to 8 to additionally include 4 diagonal directions.

[0117] Figure 8This is a conceptual diagram illustrating an example of bilateral matching. Bilateral matching is now described. A video encoder 200 or a video decoder 300 may apply bilateral matching. Bilateral matching is a technique for refining a pair of initial motion vectors MV0 and MV1. In general, bilateral matching involves the video encoder 200 or the video decoder 300 searching around MV0 and MV1 to derive refined MVs MV0' and MV1' that minimize the block matching cost. The block matching cost measures the similarity between two motion-compensated predictors generated by the two MVs (some typical criteria are: sum of absolute differences (SAD), sum of absolute transform differences (SATD), sum of squared errors (SSE), etc.) (e.g., between block 500 and block 502). The block matching cost may also include a normalization term derived based on the MV difference between the current MV pair and the initial MV pair. Certain constraints may also be applied to the MV difference between MVD0 (MV0'-MV0) and MVD1 (MV1'-MV1). Typically, the video encoder 200 or video decoder 300 may assume that MVD0 and MVD1 are proportional to the time distance (TD) between the current image and the reference image to which the two MVs point. However, in some applications, the video encoder 200 or video decoder 300 may assume that MVD0 equals – MVD1.

[0118] Decoder-side motion vector refinement (DMVR) is now described. In some examples, DMVR may include affine DMVR refinement based on sub-block offsets. In VVC, bilateral matching-based DMVR can be applied to improve the accuracy of bidirectional prediction merging candidate MVs. Bilateral matching techniques involve computing the SAD between two candidate blocks in reference image lists L0 and L1. Figure 8 As illustrated, the SAD between blocks 500 and 502 of each MV candidate around the initial MV is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bidirectional prediction signal. ¼ of the SAD value of the initial MV is subtracted from the SAD of the initial MV to be used as a regularization term. The temporal distance (e.g., POC difference) from the two reference images to the current image is the same; therefore, MVD0 has the opposite sign to MVD1.

[0119] The refinement search range is two integer luminance samples from the initial MV. The search includes an integer sample offset search stage and a fractional sample refinement stage. A full 25-point search is applied to the integer sample offset search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of the DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked according to the raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage.

[0120] The integer sample search is followed by fractional sample refinement. To reduce computational complexity, fractional sample refinement is derived using the parametric error surface equation instead of an additional search using SAD comparisons. Fractional sample refinement is conditionally invoked based on the output of the integer sample search phase. Fractional sample refinement is further applied when the integer sample search phase terminates with the minimum SAD at the center in either the first or second iteration.

[0121] In subpixel offset estimation based on parametric error surfaces, the following 2D parabolic error surface equation is fitted using the center localization cost and the cost at four neighboring locations from the center:

[0122] (1)

[0123] in( This corresponds to the fractional location with the minimum cost, and C corresponds to the minimum cost value. The above equation is solved by using the cost values ​​of the five search points. The calculation is as follows:

[0124] (2)

[0125] (3)

[0126] and The value is automatically limited to Between 8 and 8, because all cost values ​​are positive and the minimum value is This corresponds to a half-pixel offset in VVC with 1 / 16 pixel MV accuracy. The calculated score ( Added to integer distance thinning MV to obtain subpixel accurate thinning difference MV.

[0127] In VVC, the resolution of the MV is 1 / 16 of a luminance sample. An 8-tap interpolation filter is used to interpolate the samples at the fractional localization. In DMVR, the search point surrounds an initial fractional pixel MV with an integer sample offset. Therefore, for the DMVR search process, these fractionally localized samples need to be interpolated. To reduce computational complexity, a bilinear interpolation filter is used to generate the fractional samples used for the search process in DMVR. Another effect is that by using a bilinear filter with a search range of 2 samples, the DVMR process does not access more reference samples compared to the normal motion compensation process. After obtaining the refined MV using the DMVR search process, a normal 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples than in the normal motion compensation process, samples that are not needed for the interpolation process based on the original MV but are needed for the interpolation process based on the refined MV can be filled from the available samples.

[0128] When the width and / or height of the CU is greater than 16 luminance samples, the CU can be further divided into sub-blocks with a width and / or height equal to 16 luminance samples for use in the DMVR process.

[0129] In VVC, DMVR can be applied to CUs with the following modes and features: 1) CU hierarchical merging mode with bidirectional prediction MV; 2) one reference image is a past reference image and the other is a future reference image relative to the current image; 3) the distances (e.g., POC difference) from the two reference images to the current image are the same; 4) both reference images are short-term reference images; 5) the CU has more than 64 luma samples; 6) both the CU height and CU width are greater than or equal to 8 luma samples; 7) the BCW weight index indicates equal weights; 8) WP is not enabled for the current block; and / or 9) the combined inter-frame intra-frame prediction (CIIP) mode is not used for the current block.

[0130] In ECM, the conditions are simplified to: 1) CU-level merging mode or AMVP merging mode with bidirectional predictive MV; 2) Relative to the current image, one reference image is a past reference image, and the other is a future reference image; 3) The distances (e.g., POC differences) from the two reference images to the current image are the same; 4) MMVD mode is not used for the current block; 5) Neither reference image is scaled; 6) WP is not enabled for the current block; 7) CIIP mode is not used for the current block.

[0131] Figure 9 This is a conceptual diagram illustrating an example of template matching performed on a search region around the initial MV. The template matching merging pattern is now described. Template matching (TM) is a decoder-side MV derivation technique that refines the motion information of the current CU by finding the closest match between a template in the current image (e.g., the top and / or left adjacent block of the current CU) and a block in a reference image (e.g., a block of the same size as the template). Figure 9 As shown, a better motion vector (MV) is searched within the [-8, +8] pixel search range 606 in reference frame 608 around the initial motion vector 600 of the current CU 602 in the current frame 604. With AMVP candidates selected based on the initial matching error, the MVP is refined using template matching. For merge candidates indicated by the signaling merge index, their merged MVs corresponding to L0 and L1 are independently refined using template matching, and then the less precise motion vectors are further refined using the better motion vectors as priors.

[0132] like Figure 9As illustrated, a better MV is searched around the initial motion of the current CU within a search range of [-8, +8] pixels. Template matching technology from JVET-J0021 is used with the following modifications: the search step size is determined based on the AMVR mode, and in merge mode, the TM can be cascaded with the bilateral matching process.

[0133] The TM merging pattern is a combination of the regular merging pattern and template matching. Similar to the regular merging pattern, a TM merging list is first constructed, except that the MVD threshold used for the candidate pruning process differs. After constructing the TM merging list, template matching can be used to further refine each candidate in the list. The refined motion vectors can be signaled using a TM merging pattern flag and the corresponding TM merging index.

[0134] Affine mode is now described. The video encoder 200 or video decoder 300 can use affine mode. The affine motion model can be described as...

[0135]

[0136] in( ) is the coordinate ( The motion vectors at positions (a, b, c, d, e, and f) are given, and a, b, c, d, e, and f are six affine parameters. This affine motion model can be called a 6-parameter affine motion model. In a typical video decoder (e.g., a video encoder 200 or a video decoder 300), the image is divided into blocks for block-based decoding. An affine motion model for a block can also consist of three motion vectors (MVs) at three different positions not on the same line. , and To describe it. These three positions are usually called control points, and these three motion vectors are called control point motion vectors (CPMV). When the three control points are located at the three corners of the block, the affine motion can be described as...

[0137]

[0138] in, and These are the width and height of the block.

[0139] In the affine mode, different motion vectors can be derived for each pixel in a block according to an associated affine motion model. Thus, motion compensation can be performed in a per-pixel manner. However, to reduce complexity, sub-block-based motion compensation is typically employed or utilized, where the current block is divided into multiple sub-blocks (e.g., the sub-blocks can have a smaller size than the current block) and each sub-block is associated with a motion vector for block-based motion compensation. The representative coordinates of the sub-block are used to derive the motion vector of each sub-block. Generally, the center of the sub-block is used for localization. In one example, the block is divided into non-overlapping sub-blocks. If the block width is blkW, the block height is blkH, the sub-block width is sbW, and the sub-block height is sbH, then there are blkH / sbH rows of sub-blocks and blkW / sbW sub-blocks in each row. For a six-parameter affine motion model, the motion vector (referred to as the sub-block MV) for the sub-block at the i-th row (0 <= i < blkW / sbW) and the j-th (0 <= j < blkH / sbH) column is derived as

[0140]

[0141] The sub-block MV is rounded to a predefined precision and stored in a motion buffer for motion compensation and motion vector prediction.

[0142] The simplified four-parameter affine model (for scaling and rotation motions) is described as

[0143]

[0144] Similarly, the four-parameter affine model for a block can be described by two CPMVs and at two corners (usually the upper left and upper right) of the block. The motion field is then described as

[0145]

[0146] The sub-block MV at the i-th row and the j-th column is derived as

[0147]

[0148] Sub-block-based Temporal Motion Vector Prediction (SbTMVP) is now described. Similar to Temporal Motion Vector Prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located image to improve motion vector prediction and merging patterns of CUs in the current image. The same co-located image used by TMVP is also used in SbTVMP. SbTMVP differs from TMVP in two main ways: First, TMVP predicts motion at the CU level, but SbTMVP predicts motion at the sub-CU level. Second, while TMVP extracts temporal motion vectors from co-located blocks in the co-located image (e.g., the co-located block is the bottom right or center block relative to the current CU), SbTMVP applies motion shifting before extracting temporal motion information from the co-located image, where the motion shifting is obtained from the motion vector of one of the spatially adjacent blocks of the current CU.

[0149] A combined sub-block-based merge list, including SbTVMP candidates and affine merge candidates, is used for signaling delivery in the sub-block-based merge mode.

[0150] In ECM, two aspects are modified to further improve the decoding efficiency of SbTMVP. First, two co-located images are used, which are two reference frames with the minimum POC distance relative to the frame to be decoded (e.g., the current frame). Second, the motion shift of the TMVP for positioning the SbTMVP is adaptively determined from multiple locations based on template cost. More specifically, two motion shift candidate lists are constructed for the two co-located frames. The motion shift with the minimum template matching cost is used to derive the SbTMVP or TMVP candidate. The merged list based on sub-blocks includes at most four SbTMVP candidates. The SbTMVP candidate with the minimum template matching cost derived from the first co-located frame is placed in the first entry without reordering, while other SbTMVP candidates are ordered together with affine candidates. In addition, the prediction direction of each sub-block template is determined based on the central sub-block.

[0151] Figure 10 This is a conceptual diagram illustrating an example sub-block template used to generate SbTMVP. For example... Figure 10 As shown, if the central sub-block 800 is unidirectionally predicted, then all sub-block templates are unidirectionally predicted, and vice versa. If the motion vector of the corresponding adjacent sub-block at the determined reference list location cannot be used for the sub-block template, then zero MV is used for that sub-block template.

[0152] Affine merging prediction is now described. The video encoder 200 or video decoder 300 can use an affine merging mode. In the affine merging mode, the CPMV of the current CU is generated based on the motion information of spatially neighboring CUs. There may be up to five CPMV candidates, and an index is signaled to indicate which CPMV will be used for the current CU. The following three types of CPMV candidates are used to form the affine merging candidate list: 1) inherited affine merging candidates extrapolated from the CPMVs of neighboring CUs; ​​2) constructed affine merging candidate CPMVP derived using the translation MVs of neighboring CUs; ​​and 3) zero MV.

[0153] In ECM, three new types of merge candidates are introduced, and the affine merge candidate list is constructed as follows: 1) SbTMVP candidate (if available); 2) inherited from neighboring neighbors; 3) regression-based affine merge candidate; 4) affine candidate constructed from neighboring neighbors; 5) affine candidate constructed from non-neighboring neighbors; 6) history-based affine merge candidate; and 7) zero MV.

[0154] The DMVR for the affine merging pattern is now described. The video encoder 200 or video decoder 300 can use DMVR for bidirectional prediction of affine merging candidates. For example, if a candidate satisfies the DMVR criteria, a translational MV offset is added to all CPMVs of the candidate in the affine merging list. The MV offset is derived by minimizing the cost of bilateral matching, similar to regular DMVR. The difference from DMVR for bidirectional prediction of affine merging candidates is that affine motion compensation is performed to generate predictors in both directions. The MV offset search process is identical to the DMVR process in the ECM (e.g., at the predictor unit level). A 3x3 square search pattern is used to iteratively process the horizontal search range [-8, +8] and the vertical search range [-8, +8] to find the optimal integer MV offset. A half-pixel search is then performed around the optimal integer location, and error surface estimation is performed to find the MV offset with 1 / 16 accuracy.

[0155] To calculate or determine the cost of bilateral matching (for a given bidirectional affine motion), the following steps can be applied: 1) First, add offsets to each of the CPMVs in both directions to update the CPMV; 2) Apply affine motion compensation based on the updated CPMVs to generate predictors in both directions; and 3) Use a predefined cost criterion to calculate the distortion between the generated predictors.

[0156] Generally, affine motion compensation is based on sub-blocks, where the current block is divided into non-overlapping sub-blocks. Regular motion compensation is performed for each sub-block within a sub-block (e.g., all samples in a particular sub-block share the same motion). In VVC, prediction refinement using optical flow (PROF) can be applied after sub-block-based motion compensation to further improve the predictions.

[0157] U.S. Patent Application No. 18 / 351,342, filed July 12, 2023, discloses a less complex and more practical DMVR design for an affine motion model. The DMVR design example can be summarized by the following steps: 1) dividing the current block into sub-blocks; 2) generating initial motion vectors (in two predicted directions) for each sub-block (e.g., a sub-block motion field) based on the initial affine motion model; 3) iteratively processing each sub-block, calculating the sub-block bilateral matching cost for all possible offsets; 4) for each possible offset, summing the sub-block bilateral matching costs to generate the bilateral matching cost corresponding to the entire block; and 5) determining the optimal offset by selecting the offset that minimizes the bilateral matching cost corresponding to the entire block.

[0158] U.S. Patent Application No. 18 / 467,513, filed September 14, 2023, discloses a method for skipping or subsampling subblocks to further reduce complexity, wherein a subset of subblocks may be used during an affine DMVR process.

[0159] We now discuss regression-based DMVR for affine merging mode. In ECM 6.0, the linear regression-based affine merging candidate derivation technique proposed in JVET-AA0107 is adopted. In this proposal, a linear regression process is applied to derive the affine motion model, where the input to the linear regression is motion information and corresponding coordinates from a set of previously decoded sub-blocks. The video encoder 200 and video decoder 300 can be configured to perform regression-based DMVR.

[0160] The purpose of linear regression is to predict the values ​​of unknown individual dependencies using independent variables whose values ​​are known. Each predictor is weighted, with each weight representing its relative contribution to the overall prediction, and these weights are added or summed to form a prediction. The general model for multiple linear regression is Y = a + b1X1 + b2X2 + ... + b n X n Given , where Y is the dependent variable, and X1,…,X n There are n independent variables. When calculating the weights a, b1, ..., b... n When applying the least squares method, the mean square error between the statistical observation sample and the estimated value is minimized.

[0161] The affine motion model can be equivalently represented by two linear equations in matrix form as follows:

[0162]

[0163] Affine motion models are essentially linear coordinate equations with certain coefficients.

[0164] Given a set of corresponding motion vectors and coordinates N sub-blocks, linear coefficients The solution can be obtained through linear regression. Detailed information on the linear regression process can be found in U.S. Application No. 18 / 188,908 and JVET-O0066, filed March 23, 2023.

[0165] As described in the paragraph above, during the sub-block offset-based affine DMVR process, the BM cost for each sub-block is computed for all possible motion vector offsets. Therefore, the optimal sub-block motion vector offset for each sub-block within a sub-block can be determined independently and in parallel to determine the global offset for all CPMVs. This is achieved using refined motion vectors. and corresponding coordinates Linear regression can be applied to derive a new set of CPMVs as refined affine merging candidates. The detailed process that can be implemented by the video encoder 200 or the video decoder 300 is as follows: 1) Divide the decoded block into sub-blocks and generate corresponding initial sub-block MVs (e.g., generally from the initial motion information of the decoded block); 2) Perform bilateral matching for each sub-block in the sub-block to generate a set of refined MVs; and 3) The refined MVs and corresponding coordinates of these sub-blocks are used as inputs to the linear regression process, the output of which is the derived affine motion model.

[0166] Figure 11 This is a conceptual diagram illustrating a regression-based decoder with affine derivation of sign. Figure 11 The affine DMVR regression process is illustrated below. For example, video encoder 200 or video decoder 300 can determine the initial sub-block MV of sub-blocks of block 900. Video encoder 200 or video decoder 300 can apply bilateral matching to the sub-blocks to determine the refined sub-block MV, as shown in block 910. As described above, video encoder 200 or video decoder 300 can use the refined sub-block MV to apply linear regression 920 to generate an affine model.

[0167] The refinement of the affine model parameters using affine DMVR is now described. As introduced above, the affine motion model can be described as...

[0168]

[0169] The affine parameters are derived from CPMV as follows:

[0170]

[0171] When the affine merge candidates are refined as described in the paragraph above (where the same offset is added to all CPMVs), only the affine parameters are translated. Other non-translational affine parameters will be refined. , , and The process remains unchanged. In this paragraph, the affine model parameter refinement is performed using the affine DMVR as follows, and the video encoder 200 or video decoder 300 can implement this type of affine model parameter refinement.

[0172] Figures 12A to 12C This is a conceptual diagram illustrating an example of the three stages of a non-translational affine parameter search. The search process has three stages. (The remaining text appears to be incomplete and requires further context.) Figure 12A In the first stage shown, the top-left CPMV 1000 is fixed or selected as the basis MV of the affine model, and the cross mode is used to jointly search for parameters. , , and To find the value that minimizes the cost of bilateral matching. Then use , and The refined parameter values ​​are used to recalculate CPMV. In, for example... Figure 12B In the second stage shown, the new upper right CPMV 1002 is fixed or selected as the base MV, and the search parameters are combined again. , This is done to find a new value that minimizes the cost of bilateral matching, and the CPMV is recalculated after the search. In cases such as... Figure 12C In the third stage shown, the new lower left CPMV 1004 is fixed or selected as the base MV, and the parameters are adjusted again. , The same search process is applied to obtain the final refined model. For the 4-parameter affine model, only the parameters... and It needs to be refined, and the search process itself is the same as that for a 6-parameter affine model. For each search point in the search process, motion compensation can be applied to the entire CU, and the SAD between the two predictors of the CU can be used as the cost.

[0173] Figure 13This is a conceptual diagram illustrating an example independent search of CPMVs. Affine DMVR CPMV search will now be discussed. Affine DMVR CPMV search can be implemented by video encoder 200 or video decoder 300. In an affine DMVR CPMV search, each CPMV can be independently refined using regular DMVR (e.g., bilateral matching) of representative blocks. Figure 13 As illustrated, each CPMV is searched within a predefined search area based on bilateral matching of representative blocks (e.g., block 1, block 2). For example, video encoder 200 or video decoder 300 may apply an affine DMVR CPMV search to the current block 1120. A representative block may be a block that includes CPMV locations. For example, block 1100 may include CPMV locations, block 1102 may include CPMV locations, and block 1104 may include CPMV locations. Search area 1110 may represent the search area for block 1100, search area 1112 may represent the search area for block 1102, and search area 1114 may represent the search area for block 1104.

[0174] For example, a representative block could be a block centered at the CPMV location, where the coordinates of the top-left sample are (xk-halfDx, yk-halfDy), where (xk, yk) are the CPMV coordinates, halfDx is half the block width, and halfDy is half the block height. Examples of halfDx and halfDy include 2, 4, and 8. Upper bound thresholds (e.g., 8) can be set for halfDx and halfDy to limit the size of the blocks used in regular DMVR searches.

[0175] The input and output of this process are two prediction direction CPMVs, denoted as mvAffineInit[i][k] and mvAffineBest[i][k], where i=0,1 represents the index of the prediction direction, and k=0,1,2 represents the index of the CPMV. The following applies to k=0,1,2.

[0176] The video encoder 200 or video decoder 300 can use mvAffineInit[i][k] (i = 0, 1) as input to perform a regular DMVR on representative blocks k (e.g., blocks 1100, 1102, 1104) and the output is cpmv[i][k]. cpmv[0][k] = mvAffineInit[0][k] + mvOffset[k], cpmv[1][k] = mvAffineInit[1][k] - mvOffset[k].

[0177] After each CPMV is independently refined, mvAffineInit[i][k] and cpmv[i][k] are used as candidates for CPMV k, and mvAffineBest[i][k] is set as the best combination of candidates equal to the minimum two-sided matching cost of generating the current block. The following pseudocode is an example of this process:

[0178] For all i=0, 1 and k=0, 1, 2, set cpmvCand[i][k][0]=mvAffineInit[i][k] and cpmvCand[i][k][1]=cpmv[i][k].

[0179] Set minCost to equal the predefined maximum value (e.g., the maximum 64-bit integer value).

[0180] Loop from 0 to 1 through cpmvIdx0

[0181] Loop from 0 to 1 for cpmvIdx1

[0182] Loop from 0 to 1 through cpmvIdx2

[0183] For i=0, 1, cpmvTemp[i][0] is set to be equal to cpmvCand[i][0][cpmvIdx0].

[0184] For i = 0 and 1, cpmvTemp[i][1] is set to be equal to cpmvCand[i][1][cpmvIdx1]

[0185] For i=0, 1, cpmvTemp[i][2] is set to be equal to cpmvCand[i][2][cpmvIdx2]

[0186] Using cpmvTemp[i][k] (i = 0, 1, k = 0, 1, 2), compute the bilateral matching cost (denoted as costTemp) of the current block (e.g., the current block 1120).

[0187] If costTemp is less than minCost

[0188] minCost is set to equal costTemp, and mvAffineBest[i][k] is equal to cpmvTemp[i][k], i=0, 1, k=0, 1, 2.

[0189] We now discuss multi-pass decoder-side motion vector refinement in ECM. Multi-pass decoder-side motion vector refinement can be applied to either the video encoder 200 or the video decoder 300. In the first pass, bilateral matching (BM) is applied to the decoded block. In the second pass, BM is applied to each 16x16 sub-block within the decoded block. In the third pass, the motion vectors (MVs) in each 8x8 sub-block are refined by applying bidirectional optical flow (BDOF). The refined MVs are stored for spatial motion vector prediction and temporal motion vector prediction.

[0190] In the first pass, the refined MV is derived by applying BM to the decoded block. Similar to DMVR, in the bidirectional prediction operation, the refined MV is searched around the two initial MVs (MV0 and MV1) in the reference picture lists L0 and L1. Based on the minimum bilateral matching cost between the two reference blocks in L0 and L1, the refined MV (MV0_pass1 and MV1_pass1) is derived around the initial MV.

[0191] The video encoder 200 or video decoder 300 implementing bilateral matching can perform a local search to derive the integer sample precision intDeltaMV. The local search applies a 3×3 square search pattern to iterate through the horizontal search range [–sHor, sHor] and the vertical search range [–sVer, sVer], where the values ​​of sHor and sVer are determined by the block size, and the maximum value of sHor and sVer is 8.

[0192] The cost of bilateral matching is calculated as follows: bilCost = mvDistanceCost + sadCost. When the block size is cbW... When cbH is greater than 64, the Mean Removal of Absolute Difference (MRSAD) cost function is applied to remove the DC effect of distortion between reference blocks. The local search for intDeltaMV terminates when bilCost at the center point of the 3×3 search pattern has the minimum cost. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the video encoder 200 or video decoder 300 continues searching for the minimum cost until it reaches the endpoint of the search range.

[0193] We further refine the existing fractional samples to derive the final deltaMV. Then, the refined MV derivation after the first pass is as follows:

[0194] MV0_pass1 = MV0 + deltaMV

[0195] MV1_pass1 = MV1 – deltaMV

[0196] In the second pass, the refined MV is derived by applying bilateral matching to 16×16 grid sub-blocks. For each sub-block, a refined MV is searched around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass in the reference image lists L0 and L1. The refined MV (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) is derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1.

[0197] For each sub-block, the video encoder 200 or video decoder 300, which performs bilateral matching, performs a full search to derive the integer sample precision intDeltaMV. The full search has a search range of [-sHor, sHor] in the horizontal direction and a search range of [-sVer, sVer] in the vertical direction, where the values ​​of sHor and sVer are determined by the block size, and the maximum value of sHor and sVer is 8.

[0198] Figure 14 This is a conceptual diagram illustrating an example diamond-shaped region within the search area. The bilateral matching cost is calculated by applying a cost factor to the SATD cost between the two reference sub-blocks, as follows: bilCost = satdCost costFactor. Search area (2) sHor + 1) (2 sVer + 1) Divide into 5 diamond-shaped search regions 1150, such as Figure 14 As shown. A cost factor is assigned to each search region, determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond-shaped region is processed in order starting from the center of the search region. Within each region, search points are processed in raster scan order from the top left corner to the bottom right corner of the region. The minimum bilCost within the current search region is less than or equal to sbW. When the threshold of sbH is reached, the integer pixel full search is terminated; otherwise, the integer pixel full search continues to the next search area until all search points have been checked.

[0199] We further refine the existing VVC DMVR score samples to derive the final deltaMV(sbIdx2). Then, the second round of refined MV derivation is as follows:

[0200] MV0_pass2(sbIdx2) = MV0_pass1 + deltaMV(sbIdx2)

[0201] MV1_pass2(sbIdx2) = MV1_pass1 – deltaMV(sbIdx2)

[0202] In the third pass, the refined MV is derived by applying BDOF to 8×8 grid sub-blocks. For each 8×8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy, without truncating from the refined MV of the parent-child blocks in the second pass. The derived bioMv(Vx, Vy) is rounded to 1 / 16 sample precision and truncated to between -32 and 32.

[0203] The third, refined MV (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) derivation is as follows:

[0204] MV0_pass3(sbIdx3) = MV0_pass2(sbIdx2) + bioMv

[0205] MV1_pass3(sbIdx3) = MV0_pass2(sbIdx2) – bioMv

[0206] Bidirectional optical flow (BDOF) is now described. BDOF is used to refine the bidirectional prediction signal of luminance samples in a CU at a 4×4 sub-block level. Video encoder 200 and video decoder 300 can use BDOF to refine the bidirectional prediction signal of luminance samples in the CU at a 4×4 sub-block level. BDOF is based on the concept of optical flow, which assumes that the motion of the object is stationary. For each 4×4 sub-block, motion refinement is calculated by minimizing the difference between the L0 and L1 prediction samples. Then, motion refinement is used to adjust the bidirectional prediction sample values ​​in the 4x4 sub-blocks. The following steps are applied during the BDOF process.

[0207] First, the horizontal and vertical gradients of the two predicted signals. and ( This is done by directly calculating the difference between two adjacent samples, for example,

[0208]

[0209] in It is a list ( Coordinates of the predicted signal in ) The sample value at that location, and shift1 is calculated based on the luminance bit depth bitDepth, because shift1 is set to equal to 6.

[0210] Then, gradient , , , and The autocorrelation and cross-correlation are calculated as follows:

[0211] in

[0212]

[0213] in It is a 6×6 window surrounding a 4×4 sub-block, with shift2 set to 4 and shift3 set to 1.

[0214] Then, using cross-correlation and autocorrelation terms, the motion refinement is derived using the following formula. :

[0215]

[0216] in . It is the floor function.

[0217] Based on motion refinement and gradients, the following adjustments are calculated for each sample in the 4×4 sub-block:

[0218]

[0219] Finally, the BDOF samples of CU are calculated by adjusting the bidirectional prediction samples as follows:

[0220]

[0221] Specifically, setting shift5 to equal Max(3, 15) BitDepth), and variables Set to equal to (1 << (shift5) 1)).

[0222] These values ​​were chosen to ensure that the multiplier in the BDOF process does not exceed 15 bits, and that the maximum bit width of the intermediate parameters in the BDOF process remains within 32 bits.

[0223] Figure 15 This is a conceptual diagram illustrating an example extended CU region used in BDOF. For example... Figure 15 As shown, the BDOF procedure uses an extended row and column around the CU boundary. Figure 15In the diagram, the CU is shown as a 4x4 block 1200, and an extended row and column are shown as a 6x6 surrounding region 1202. To control the computational complexity of generating prediction samples outside the boundary, prediction samples in the extended region (white positioning) are generated by directly taking reference samples at nearby integer locations (using floor() operations on the coordinates) without interpolation, and prediction samples within the CU (grayscale positioning) are generated using a standard 8-tap motion-compensated interpolation filter. These extended sample values ​​are used only for gradient calculation. For the remaining steps in the BDOF process, if any samples and gradient values ​​outside the CU boundary are needed, those values ​​are filled in from the nearest neighbor (e.g., repeated).

[0224] The video encoder 200 and video decoder 300 can use the BDOF process to refine the bidirectional prediction signal of the CU at a 4×4 sub-block level. BDOF can be applied to the CU if the following conditions are met: 1) The CU is decoded using a “true” bidirectional prediction mode, for example, one of two reference images is displayed before the current image and the other is displayed after the current image; 2) The CU is not decoded using an affine mode or an ATMVP merging mode; 3) The CU has more than 64 luma samples; 4) Both the CU height and CU width are greater than or equal to 8 luma samples; 5) The BCW weight index indicates equal weights; 6) WP is not enabled for the current CU; and 7) CIIP mode is not used for the current CU.

[0225] JVET-AE0065 proposes an "Iterative BDOF pass in multi-pass DMVR" technique, which is included in the ECM. This technique involves extending the multi-pass DMVR by adding a fourth pass of BDOF MV refinement. In this extended technique, the maximum sub-block size of the third pass of BDOF MV refinement is always 8x8, and when the decoded block size is less than 1024, the maximum sub-block size of the fourth pass of BDOF MV refinement is adaptively set to 4x4; otherwise, it is set to 8x8. The fourth pass of BDOF MV refinement is skipped when the sub-block of the third pass of BDOF MV refinement derives differential motion with Vx equal to 0 and Vy equal to 0.

[0226] Currently in ECM, for merge modes, the LIC flag and BCW index are always inherited from neighboring blocks. For example, for the current block, the video decoder 300 can inherit the LIC flag and BCW index from neighboring blocks. However, this may not be the optimal solution, as lighting changes may occur in neighboring blocks but not in the current block, or vice versa. To address this issue, this paper describes techniques for deriving the LIC flag and / or BCW index for merge modes. Considering the various merge modes and the applicability of LIC to these modes, this paper describes the interaction between the LIC flag and regular merge mode, TM merge mode, and affine merge mode.

[0227] The disclosed techniques may be used individually (e.g., separately) or in any combination.

[0228] In some examples, the video encoder 200 or video decoder 300 may always disable LIC for merge mode. Since LIC can be applied not only to merge mode but also to AMVP mode, there is already an option to enable or disable LIC technology in AMVP mode. Therefore, for merge mode, to reduce encoding and decoding complexity, the LIC flag can always be set to "false," regardless of the LIC flag values ​​of adjacent blocks.

[0229] In other examples, the video encoder 200 or video decoder 300 may always enable LIC for merge mode. When LIC is executed, the templates of the current block and the reference block can be used as inputs to derive the scaling factor and offset through a mean squared error (MSE) minimization process. If there is no illumination change between the current template and the reference template, the scaling factor and offset may be close to 1 and 0, respectively, so that even if the video encoder 200 or video decoder 300 applies the derived LIC mode, there may be little impact on the motion compensation results compared to not applying LIC. However, when there is an illumination change between the current template and the reference template, the derived LIC model can help improve the motion compensation results, which may be beneficial. Therefore, for merge mode, the LIC flag can be set to "true" regardless of the LIC flag values ​​of adjacent blocks.

[0230] In some examples, video encoder 200 may signal the LIC flag used for merge mode. In such examples, video decoder 300 may obtain the LIC flag in the bitstream and parse it to determine whether the LIC flag is "true" or "false" (e.g., having a value equal to 1 or 0). Because the LIC model can be derived based on the current template and a reference template and applied to the current decoded block, the LIC model may be inaccurate because the template may not be a good representation of the current block and the reference block. Therefore, another option is to signal the LIC flag used for merge mode and have the encoder use bit rate distortion (RD) to determine the LIC flag value, rather than having video encoder 200 or video decoder 300 inherit the LIC flag from neighboring blocks. For example, video encoder 200 may use RD to determine the LIC flag value and signal this LIC flag value to video decoder 300 in the bitstream.

[0231] In one example, a separate flag is used to indicate whether the LIC flag used for merge mode is always on or off, for example, regardless of the LIC flag values ​​of adjacent blocks. For example, video encoder 200 may signal this separate flag in the bit stream, and video decoder 300 may obtain the flag in the bit stream and parse the flag to determine whether the LIC flag used for merge mode is always on or off.

[0232] In another technique, separate flags are used to indicate whether the inherited LIC flag has been inverted. Therefore, the final LIC flag is determined together with the inherited flag and the signaled flag. For example, if the inherited flag is not equal to the signaled flag, the final LIC flag is "true"; otherwise, the final LIC flag is "false".

[0233] In some examples, video decoder 300 may determine to decode the current block of video data using a merging mode. Video decoder 300 may obtain a flag from the bitstream. Based on the value of the flag, video decoder 300 may determine to use a second merge list from two merge lists for the current block. The second merge list is based on the first merge list from the two merge lists. Video decoder 300 may decode the current block based on the second merge list.

[0234] For example, for a merge mode, in order to signal the LIC flag for, for example, a regular merge mode, a second regular merge list can be created, for example, by copying the original regular merge list and inverting the LIC flag value of each entry, since in the merge mode, one of the following LIC flag variants can be tried. As used herein, inverting or flipping the LIC flag value means changing the LIC flag value to the opposite value, such as from "true" to "false", from "false" to "true", 1 to 0, 0 to 1, etc. The video decoder 300 can generate a first merge list (e.g., the original regular merge list), which includes identifying a plurality of merge candidates, each of the plurality of merge candidates in the first merge list having a corresponding LIC flag value inherited from the corresponding adjacent block. The video decoder 300 can also generate a second merge list, which includes copying the plurality of merge candidates from the first merge list to the second merge list and changing the corresponding LIC flag value of each of the plurality of merge candidates to the opposite value.

[0235] The new conventional merge list may compete with other existing merge modes through the SATD and RD stages. When a conventional merge mode is selected, the video encoder 200 may signal additional mode flags to the video decoder 300 to indicate whether to use the original conventional merge list or the new conventional merge list. For example, the video encoder 200 may signal additional mode flags to the video decoder 300 in the bitstream to indicate whether to use the original conventional merge list or the new conventional merge list. The video decoder 300 may parse the additional mode flags to determine the values ​​of the additional mode flags indicating which of the two merge lists to use. In some examples, the video decoder 300 may generate a first merge list (e.g., a conventional merge list) and generate a second merge list (e.g., a new conventional merge list) based on the first merge list. For other merge mode-related techniques (e.g., TM merge mode and / or affine merge mode), a scheme similar to that described for the conventional merge mode may be used to signal the LIC flags to be used. In one example, in one merge list, all LIC flags may be set to "false" without inheritance, and for another merge list, all LIC flags may be set to "true".

[0236] For affine merging modes, the list may include not only affine merging candidates but also SbTMVP merging candidates. For example, video encoder 200 or video decoder 300 may include both SbTMVP and affine merging candidates in the candidate list. For instance, a first merge list may include multiple candidates, where the multiple candidates include at least one affine merging candidate and at least one SbTMVP candidate. Since SbTMVP and affine merging candidates are very different, it may be reasonable to exclude SbTMVP candidates from LIC flag signaling. In one example, when constructing a new affine merging list (e.g., similar to or related to an example with a signaled LIC flag), SbTMVP candidates are excluded from the process before copying the original affine merging list and inverting the LIC flag. For example, video decoder 300 may generate a second merge list based on the first merge list, which includes copying multiple candidates from the first merge list to the second merge list, inverting the LIC flags for each corresponding candidate in the second merge list, and excluding at least one SbTMVP candidate such that the second merge list does not include at least one SbTMVP candidate. In another example, an SbTMVP candidate can be replaced by other affine merge candidates that have not yet been inserted into the list due to list size limitations. For example, as part of generating a second merge list, the video decoder 300 can replace at least one SbTMVP candidate with at least one other affine merge candidate.

[0237] In another example, when the LIC flag is signaled, a candidate list for this mode can be derived differently from the regular merge candidate list. In one example, only a few candidates preceding the regular merge candidates can be used to generate the candidate list. For example, when the LIC flag is signaled, the video encoder 200 or the video decoder 300 can use a subset of the regular merge candidates from the candidate list.

[0238] In some examples, video decoder 300 may determine to decode the current block of video data using a merging mode. Video decoder 300 may generate a first merging list for the current block, wherein generating the first merging list includes applying template matching to candidates in the first merging list. Video decoder 300 may generate a second merging list based on the first merging list. Video decoder 300 may use a merging mode and decode the current block based on either the first or second merging list. For example, when the LIC flag is signaled to be equal to a "true" value, the second candidate list for this mode is derived by flipping or reversing all inherited LIC flags from the regular candidate list, and is trimmed before or after a reordering based on template matching cost (e.g., adaptive reordering of merging candidates (ARMC) or as discussed below). In some examples, as part of generating the second merging list, video decoder 300 may trim the candidates in the first merging list for the second merging list before applying template matching to the candidates in the first merging list. In some examples, video decoder 300 may trim multiple merging candidates in the second merging list while applying template matching to the candidates in the first merging list. For example, when the video encoder 200 signals the LIC flag, the video decoder 300 can deduce the candidate list by changing the inherited LIC flag from the regular candidate list, for example, from 1 to 0 and / or from 0 to 1. When generating the candidate list, not all candidates are necessarily used. In one example, only the first N candidates from the regular merge list are used. Candidate list generation can also occur after a reordering based on template matching cost. In one example, only the first N candidates after the ARMC process are used to generate a new merge list. For example, the video decoder 300 can trim the candidate list.

[0239] When flipping the LIC flag, it is not simply a matter of flipping or reversing the LIC flag from "true" to "false" or vice versa. There are more options available because, in some examples, multiple sets of LIC parameters can be stored in the neighboring CUs derived from the original merge candidate, as described above.

[0240] When the LIC flag of the original merge candidate is equal to "false", instead of flipping the LIC flag from "false" to "true" and deriving the LIC parameter set using the template of the current block and the template of the reference block, there may also be a set of LIC parameters derived using NLIC technology and stored in the corresponding adjacent CU. Therefore, another possibility is to flip or reverse the LIC flag to "true" and use the NLIC parameter set stored in the adjacent CU. In this way, when the adjacent block has a LIC flag equal to "false", two candidates can be generated instead of one. For example, when the adjacent block has a LIC flag equal to "false", the video encoder 200 or the video decoder 300 can generate two candidates. For example, the video decoder 300 can generate a second merge list, which includes changing a LIC flag value associated with the first merge list from "false" to "true" and using an NLIC parameter set of the adjacent decoding unit to generate two candidates.

[0241] Similarly, when the original merge candidate has a LIC flag equal to "true", a total of three options are available. The first option is to simply flip or reverse the LIC flag to "false"; the second option is to keep the LIC flag equal to "true" and inherit the set of LIC parameters (derived between the template of the adjacent CU and the reference template of the adjacent CU) stored in the corresponding neighboring CU. The third option still keeps the LIC flag equal to "true", but inherits the NLIC parameters (derived between the reconstructed sample and the predicted sample) if they are available from the neighboring CU. For example, video encoder 200 or video decoder 300 can employ any of these three options. For example, as part of generating the second merge list, video decoder 300 can determine that the value of the Local Illumination Compensation (LIC) flag associated with the first merge list is "true". Based on the determination that the value of the LIC flag associated with the first merge list is "true", the video decoder 300 may: a) change the value of the LIC flag associated with the first merge list to "false" for use in the second merge list; b) keep the value of the LIC flag associated with the first merge list as "true" for use in the second merge list, and inherit a set of LIC parameters from an adjacent CU; or c) keep the value of the LIC flag associated with the first merge list as "true" for use in the second merge list and inherit a set of available nonlocal illumination compensation (NLIC) parameters from an adjacent CU.

[0242] If the NLIC mode is used to decode the original merge candidate, a total of three options are available. The first option is to set the LIC flag to "false"; the second option is to simply set the LIC flag to "true" and derive the LIC parameters using the template of the current block and the reference template of the current block; the third option is available if the adjacent CU stores a set of LIC parameters, in which case the LIC parameters can be inherited and the LIC flag can be set to "true" for the new candidate. For example, the video encoder 200 or the video decoder 300 can use any of these three options.

[0243] The LIC parameter set can include two parameters, one representing the slope and the other representing the offset. Therefore, instead of directly inheriting the LIC parameter set from adjacent blocks, one can inherit and adjust it. In one example, the LIC parameter set is inherited and used in conjunction with slope adjustment. In a second example, the LIC parameter set is inherited and used in conjunction with offset adjustment. In a final example, the LIC parameter set is inherited and used in conjunction with both slope and offset adjustments. For example, for the current block, video encoder 200 or video decoder 300 can inherit the LIC parameter set from adjacent blocks and at least use slope adjustment and / or offset adjustment. For example, as part of generating the second merge list, video decoder 300 can inherit the LIC parameter set from adjacent CUs and adjust at least one of the slope parameter or the offset parameter of the LIC parameter set.

[0244] Because of the LIC flag derivation technique based on template matching cost proposed in JVET-AF0128, different combinations of techniques can be designed. In one example, the LIC flag can be flipped or reversed before applying the template matching technique to create a new merge list. In this way, the new list will have the flipped or reversed LIC flag, while the original list will have the LIC flag determined by the template matching cost technique. For example, video encoder 200 or video decoder 300 can flip or reverse the LIC flag to generate a new candidate list before applying the template matching technique. For example, as part of generating a second merge list, video decoder 300 can change the values ​​of several LIC flags associated with the first merge list to the opposite values ​​for the second merge list before applying template matching to the candidates of the first merge list.

[0245] In the second example, the LIC flag can be flipped or reversed after the template matching cost-based LIC flag derivation technique, and the template matching cost-based technique can modify the LIC flag values ​​in the original merge list. For example, as part of generating the second merge list, the video decoder 300 can change multiple LIC flag values ​​associated with the first merge list to opposite values ​​for the second merge list after applying template matching to candidates of the first merge list.

[0246] Regardless of the decision made by the technique based on template matching cost, the LIC flag value can be flipped or reversed in the new merge list. For example, video encoder 200 or video decoder 300 can generate such a new candidate list.

[0247] In the third example, because the template matching cost-based LIC flag derivation technique is currently only applied to unidirectional predictive merge candidates, LIC flag flipping or inversion can be applied only to bidirectional predictive merge candidates, such that the new merge candidate list includes only bidirectional predictive merge candidates from the original merge list with inverted LIC flag values. For example, video encoder 200 or video decoder 300 can only flip or invert the LIC flags used for bidirectional predictive merge candidates when generating a new merge candidate list. For example, video decoder 300 can determine that one or more candidates in the first merge list are bidirectional predictive merge candidates. For each bidirectional predictive merge candidate in the bidirectional predictive merge candidate list, video decoder 300 can change the corresponding LIC flag value associated with the first merge list to the opposite LIC flag value for the second merge list.

[0248] In the fourth example, a switching criterion is defined to switch between a template-matching cost-based LIC flag derivation technique and a LIC flag flipping technique. For example, a block size threshold N can be defined, where N represents the number of samples in the current block. If the number of samples is greater than N, the template-matching cost-based LIC flag derivation technique is applied; otherwise, the flipping and flag signaling transmission technique is applied, or vice versa. For example, video encoder 200 or video decoder 300 can apply the block size threshold N. For example, video decoder 300 can determine whether the block size of the current block is greater than a threshold. Based on the determination that the block size of the current block is greater than the threshold, video decoder 300 can a) determine one or more LIC flag values ​​for the second merge list based on template-matching cost; or b) determine one or more LIC flag values ​​for the second merge list by changing the corresponding values ​​of one or more LIC flags in the first merge list for the second merge list.

[0249] In another example, the template-matching cost-based LIC flag derivation technique is simply disabled, and only the flip and flag signaling delivery techniques are used. For example, the video encoder 200 or the video decoder 300 may not apply the template-matching cost-based LIC flag derivation technique and may only apply the flip and flag signaling delivery techniques described herein.

[0250] We will now discuss conditional signaling delivery of the LIC flag. For merge mode, signaling delivery of the LIC flag may not always be applied. In some cases, the LIC flag can still be inherited from neighboring blocks.

[0251] The block size can be used to determine whether the LIC flag is signaled or inherited from neighboring blocks. In one example, if a block has fewer than N samples, the LIC flag is inherited and not signaled. In a second example, if a block has more than M samples, the LIC flag is inherited and not signaled. In a third example, two thresholds from the first two examples are applied, and the LIC flag is signaled only if the block has more than N and less than M samples. For example, a video encoder 200 or a video decoder 300 could employ this technique.

[0252] Time IDs can also be used as criteria. In ECMs in a random access configuration, time IDs are assigned to each image, and generally, a higher time ID indicates that the reference image of the image is located closer in time distance than if the time ID is lower. Closer time distances generally indicate stronger similarity between the image and the reference image, and therefore, images with higher time IDs can usually be decoded relatively cheaply with a small number of bits. In this case, the overhead of signaling the LIC flag for images with higher time IDs may be relatively large. Based on the above statements, in one example, the LIC flag of images with time IDs greater than K is inherited, otherwise the LIC flag is signaled. For example, if the current image has a time ID greater than K, the video encoder 200 or video decoder 300 can inherit the LIC flag of blocks of the current image. If the current image does not have a time ID greater than K, the video encoder 200 can signal the LIC flag of blocks of the current image in the bitstream, and the video decoder 300 can parse the LIC flag of such blocks.

[0253] Block position is another piece of information that can be used to determine whether the LIC flag should be signaled. For example, when the block is located at the top left corner of the image, there may be no available template, so the LIC model is not derived and the LIC flag is set to "false". In this case, there is no need to signal the LIC flag. When the block is located at the top or left edge of the image, only the left or top template is available. In this case, the video encoder 200 may also skip the signaling of the LIC flag. In summary, in one example, when the top left corner coordinates of the block have a horizontal coordinate x = 0 or a vertical coordinate y = 0, the LIC flag may not be signaled. For example, if the top left corner coordinates of the block have a horizontal coordinate x = 0 or a vertical coordinate y = 0, the video encoder 200 may avoid signaling the LIC flag of the block.

[0254] Now let's discuss context separation. CABAC contexts can be applied when decoding a signaled LIC flag. In one example, two contexts are used: the first context is used if either the left or upper neighbor of the current block is decoded in AMVP mode and the signaled LIC flag is "true"; otherwise, the second context is used. In another example, three contexts are used (the context indices are derived via leftIsLIC + aboveIsLIC), where leftIsLIC is one if the left neighbor exists and is decoded in AMVP mode, and the LIC flag of the left neighbor is on; and aboveIsLIC is one if the upper neighbor exists and is decoded in AMVP mode, and the LIC flag of the upper left neighbor is on. In yet another example, only one context is used. In yet another example, the same context is used for decoding the LIC flag in AMVP mode. For example, a video encoder 200 can employ any of these techniques.

[0255] Block size can also be used to separate contexts. In one example, different contexts are used to signal the LIC flag, depending on the block size of the current block. For example, if the number of samples in the current block is less than N, one context is used; if the number of samples in the current block is greater than N, a second context is used. For example, video encoder 200 can determine whether the number of samples in the current block is less than N, and use a specific context based on this determination.

[0256] In some examples, the video encoder 200 or video decoder 300 can reorder the candidate list. The candidate list can be reordered based on cost (e.g., template matching cost). When LIC is enabled, the cost can be derived by considering the LIC model for each candidate. However, this can increase complexity because different candidates may have different LIC models that need to be derived. To avoid this complexity, candidate reordering can be performed without applying the LIC model, and the LIC model can be applied only during the prediction phase. Furthermore, cost metrics should be aligned (e.g., using only one of SAD, MRSAD, SATD, sum of absolute transform differences with mean removed (MRSATD), SSE, sum of squared errors with mean removed (MRSSE)). In this way, the reordering portion of the LIC model with the signaled LIC flag and the regular merging pattern can be shared.

[0257] In some examples, video encoder 200 or video decoder 300 can influence the interaction between LIC and DMVR. For regular merging mode, affine merging mode, and TM merging mode, DMVR can be applied if the candidate meets the DMVR criterion. For example, video decoder 300 can determine to apply DMVR based on determining that one or more DMVR conditions are met. Currently in ECM, if the LIC flag of a merge candidate is "true", only LIC can be applied, and DMVR is skipped, even if other DMVR conditions are met. Therefore, current ECM does not allow the use of LIC and DMVR together. However, since both techniques can refine the predicted signal, enabling LIC and DMVR simultaneously can be beneficial. In one example, if a merge candidate meets other DMVR conditions besides the current LIC flag condition (e.g., where the LIC flag value is "true"), both the DMVR and LIC processes can be applied. For example, video decoder 300 can determine to decode the current block of video data using the merging mode. Video decoder 300 can determine to apply LIC to the current block. Video decoder 300 can determine to apply DMVR to the current block. The video decoder 300 can decode the current block based on the application LIC and the application DMVR. For example, the video decoder 300 can obtain the LIC flag in the bitstream. As part of determining the application LIC, the video decoder 300 can determine the value of the LIC flag, and the value of the LIC flag indicates the application LIC.

[0258] In the second example, contrary to the current ECM design, if the merge candidate meets other DMVR conditions and the LIC flag is still "true", then DMVR can be applied and LIC can be bypassed. In the third example, compared to the second example, the LIC flag can be additionally set to "false" for future inheritance. For example, video encoder 200 or video decoder 300 can utilize either of these techniques.

[0259] Currently, when performing DMVR with the LIC flag set to "false", SAD is used as a distortion metric during the refinement process. As mentioned in the first example above, performing DMVR with the LIC flag set to "true" is still feasible. However, due to lighting variations, using SAD with the mean removed (MRSAD) (instead of SAD) as a distortion metric may be more accurate. In one example, when performing DMVR for affine merge mode with the LIC flag set to "true", MRSAD is used instead of SAD. For example, video encoder 200 or video decoder 300 may use MRSAD as a distortion metric during the DMVR refinement process. For example, video decoder 300 may use MRSAD as part of applying DMVR. For example, video decoder 300 may use MRSAD instead of SAD as part of using MRSAD.

[0260] For affine DMVR, a series of sequential processing steps (as described above) can be applied, which introduces complexity, especially on the decoder side, since DMVR is a decoder-side refinement technique. When affine DMVR is allowed to be performed with the LIC flag set to "true," the complexity of both LIC and affine DMVR in the video decoder can increase. To reduce the complexity of the combination of LIC and affine DMVR, the affine DMVR process can be performed as a simplified version. For example, the video decoder 300 can perform LIC and a simplified version of DMVR. In one example, when the LIC flag is "true," affine DMVR is applied only with the sub-block offset-based affine DMVR refinement technique. For example, the video decoder 300 can apply only sub-block offset-based affine DMVR refinement. In a second example, affine DMVR is applied for both the sub-block offset-based affine DMVR refinement technique and the regression-based affine DMVR refinement technique. For example, the video decoder 300 can apply both sub-block offset-based affine DMVR refinement and regression-based affine DMVR refinement. In the third example, affine DMVR is performed using all DMVR steps, except for the affine model parameter refinement technique. For example, the video decoder 300 can apply affine DMVR and bypass affine model parameter refinement.

[0261] In some examples, the video encoder 200 or video decoder 300 can utilize LIC on / off (e.g., true / false) switching without signaling transmission. As described above, to switch the LIC flag for the merge mode on / off, a second merge list can be created from the original list, and a signal flag can be sent to indicate whether the original merge list or the new merge list should be used. However, saving signaling overhead is feasible. In one example, the original list and the new list are merged into a single list via concatenation, and the merged list is used as input to the SATD stage. In a second example, the two merge lists are merged in an interleaved order instead of concatenating, compared to the first example.

[0262] In some examples, the video encoder 200 or video decoder 300 can select the BCW weights. When a new merge list is created for LIC flag signaling of the merge mode, the BCW weights can be handled differently compared to directly copying the BCW weights from the original merge list. In one example, the BCW weights in the new merge list are set to preset values. In ECM, a template-matching-based BCW weight derivation technique is introduced. Basically, in the case of bidirectional prediction of merge candidates, two reference templates are combined into one reference template using different BCW weights, and the optimal BCW weights are determined by comparing the SAD cost with the template of the current block. When the LIC flag is "true", the derived LIC model is appended before the SAD calculation and applied to the current template, which will lead to different results compared to when the LIC flag is "false". Therefore, in the second example, the BCW weights of the new merge list are independently derived from the original merge list.

[0263] In some examples, the video encoder 200 or the video decoder 300 may use a LIC model to derive parameters for weighted prediction. This disclosure describes the use of a LIC model to derive parameters for weighted prediction when LIC is enabled.

[0264] For example, in ECM-10.0, LIC can be applied to bidirectional prediction. In this case, LIC can be applied to each predictor before calculating a weighted average of the two predictors as the final bidirectional predictor. In this example, the weights used in the weighted averaging process can be derived using the LIC model parameters and the reconstructed samples available when reconstructing the current block. For example, a video encoder 200 or a video decoder 300 can use such a technique.

[0265] For example, the same row sample used to derive LIC parameters (called the "template sample"), and the template of the current block (called the "current template") The template of the reference block is called the "reference template". The term "subsampling" is used to derive weights. For example, the template in LIC model derivation can be subsampled, for example, not all samples in the template can be used. Therefore, using the mentioned subsampling, deriving the model's cost using the entire template might be more accurate, as the model can be validated on samples that are not used in the model derivation. In one example, whether or not subsampling is used can be used as a criterion for validating the model, for example, to derive the cost with respect to the template. If subsampling is not used, other samples not included in the model derivation can be used for cost derivation. For example, a video encoder 200 or a video decoder 300 can use such a technique.

[0266] For example, for each predictor, the LIC parameter is applied to and the SAD between the obtained sample and the current template ( When performing a weighted average, the weights of the two predictors are calculated as follows: ,in This corresponds to applying the corresponding LIC parameter to and The calculated cost. It is used for Weights. For example, a video encoder 200 or a video decoder 300 can use such techniques.

[0267] Specific conditions can be designed to turn the proposed technology on or off.

[0268] As an example, when At that time, the proposed technology was shut down. As another example, when... and / or At this time, the proposed technology is turned off. As yet another example, one or more syntax elements are signaled to indicate whether the proposer is used, and this signaling can be at different levels in the bit stream (SPS, PPS, Slice, CTU, CU, etc.). For example, a video encoder 200 or a video decoder 300 may use such a technology.

[0269] The disclosed techniques can be used as part of the signaled notification for BCW. As an example, in the design of BCW signaling after LIC, the disclosed techniques can be used as part of the signaled notification for BCW. Depending on the value of the LIC syntax, BCW signaling can be used as follows: If LIC == 0, BCW signaling remains unchanged; if LIC != 0, BCW signaling has two options: a) use the default BCW (usually a 1:1 weighted average); or b) use the disclosed weighted average based on LIC cost. As yet another example, in the design of LIC signaling after BCW, the disclosed techniques can be used as part of the signaled notification for BCW. Depending on the value of the BCW syntax, LIC can be signaled as follows: If the BCW syntax element has a value of 0: signal LIC, and use the default BCW for both LIC == 0 and LIC != 0; if the BCW syntax element is another specified value (e.g., 1), then signal LIC. If LIC is 0, the BCW weights corresponding to the BCW syntax element values ​​are used; if LIC is non-zero, the techniques disclosed in this disclosure are used without using BCW weights; if the BCW syntax element is any other value, the LIC flag is not signaled, and the value of the LIC flag is set to 0. For example, video encoder 200 or video decoder 300 may use such techniques.

[0270] In one example, video encoder 200 or video decoder 300 may apply BDOF and / or bilateral DMVR (BDMVR) to a block decoded by LIC. In the current ECM, the decoded block may apply PROF to refine the luma prediction block, and the PROF-refined luma prediction block may be further refined by applying LIC. However, BDOF refinement is not applied to the block when it meets the BDOF condition and is determined to be refined by applying LIC.

[0271] This disclosure describes the application of LIC refinement and BDOF refinement to a decoded block when the decoded block is determined to be to be applied to (e.g., by signaling a LIC flag or deriving a LIC flag from a merged candidate) and the decoded block satisfies the BDOF condition. The BDOF condition is one or more of the following: 1) the current block is a bidirectional predictive decoded block; 2) the current block has two reference blocks, wherein one reference block is located in a past reference picture relative to the current picture, and the other reference block is located in a future reference picture relative to the current picture; 3) the current block has a POC distance equal to that of the two reference pictures relative to the current picture; 4) the current block has an equal weight for each reference block. For example, a video encoder 200 or a video decoder 300 may use such a technique.

[0272] This disclosure describes applying LIC refinement and bilateral DMVR (BDMVR) refinement to a decoded block when the decoded block is determined to be subject to LIC (e.g., a signaling LIC flag or a LIC flag deduced from a merge candidate) and the decoded block satisfies the BDMVR condition. The BDMVR condition is one or more of the following: a) the current block is a bidirectional predictive decoded block; b) the current block has two reference blocks, wherein one reference block is located in a past reference picture relative to the current picture, and the other reference block is located in a future reference picture relative to the current picture; c) the current block has a POC distance equidistant from the current picture from both reference pictures. For example, a video encoder 200 or a video decoder 300 may use such a technique.

[0273] In one example, when both LIC and BDOF are applied to the decoded block, BDOF sub-block MV refinement is skipped. BDOF is applied to adjust the luminance samples. In one example, for each luminance sample, the video encoder 200 or video decoder 300 derives Vx and Vy, which are then applied to the gradient of the sample to derive the luminance sample adjustment value. In another example, a subset of luminance samples (e.g., a sub-block MxN of the decoded block) shares the same Vx and Vy, which are applied to the gradient of each luminance sample within the MxN sub-block to derive the luminance sample adjustment value.

[0274] In one example, when both LIC and BDMVR are applied to the decoded block, BDMVR sub-block MV refinement is skipped. BDMVR is only applied to refine MV0 and MV1 at the decoded block level. LIC is applied using the MV refined at the PU level. In the case of affine-decoded blocks, BDMVR is only applied to CPMV refinement at the decoded block level. The updated CPMV is used to derive the updated sub-block motion vector field, and the updated sub-block motion vector field is used in the LIC process. For example, a video encoder 200 or a video decoder 300 can use this technique.

[0275] In one example, when both LIC and BDMVR are applied to a decoded block, BDMVR is still applied to the decoded block level; however, LIC uses the unrefined decoded block level MV instead of the refined MV for model derivation. Similarly, in the case of affine-decoded blocks, the unrefined CPMV will be used for LIC. For example, video encoder 200 or video decoder 300 can use this technique.

[0276] In one example, when both LIC and BDMVR are applied to the decode block, not only is the decode block-level BDMVR applied, but also the sub-block-level BDMVR refinement is applied. However, the MV refined at the PU level is used in the LIC process. For example, a video encoder 200 or a video decoder 300 can use this technique.

[0277] In yet another example, based on the example above, the original, unrefined MV (rather than the PU-level refined MV) is used in the LIC process.

[0278] In one example, when both LIC and BDMVR are applied to a decoding block, not only is BDOF luminance sample adjustment applied, but also BDOF sub-block MV thinning is applied. The LIC process is applied after BDOF and together with the BDOF thinned sub-block MV. For example, a video encoder 200 or a video decoder 300 can use this technique.

[0279] In another example, based on the example above, the LIC process is applied before BDOF, and the sub-block MV used in the LIC is the input to the BDOF process.

[0280] In one example, when the LIC is determined to be applied to a decoded block and the decoded block satisfies the BDOF and / or BDMVR conditions, the entire decoded block has MV0 on the L0 reference picture and MV1 on the L1 reference picture. MV0 and MV1 have minimum bilateral matching costs for local surrounding regions, where the local surrounding region is defined using the center MV on each L0 and L1 reference picture, and the center MV is signaled from the merge candidate derivation or in the bitstream. For example, a video encoder 200 or a video decoder 300 can use such a technique.

[0281] In one example, the LIC reconstruction template is the adjacent top row / left column of the decoded block, and the LIC reference template is derived from MV0 and MV1 with offsets (-1, -1) on reference images L0 and L1, respectively. Here, MV0 and MV1 can be the MV refined by BDMVR. For example, a video encoder 200 or a video decoder 300 can use this technique.

[0282] In one example, the LIC reconstruction template is the adjacent top / left row with an offset of K at the top / left boundary of the decoded block, and the LIC reference template is derived from MV0 and MV1 with offsets (-K - 1, -K - 1) on reference images L0 and L1, respectively. Here, K is the BDOF extended reference block size, K <= 3. MV0 and MV1 can be the MV refined by BDMVR. For example, a video encoder 200 or a video decoder 300 can use this technique.

[0283] In one example, the LIC reference templates and LIC reconstruction templates for L0 and L1 are used separately to derive the LIC parameters for L0 and L1. The derived LIC parameters for L0 and L1 are applied to the BDOF extended reference blocks for L0 and L1, respectively, and the LIC-refined extended reference blocks are used as inputs to the BDOF process to derive gradients and Vx, Vy, etc., which are used to derive luminance sample adjustments. The luminance sample adjustments are applied to weighted bidirectional prediction to derive the final prediction block. For example, a video encoder 200 or a video decoder 300 can use this technique.

[0284] In one example, the LIC reference templates of L0 and L1 are used together to derive the LIC parameters of the decoded block. The derived LIC parameters are applied to BDOF and / or BMDVR to refine the prediction block to derive the final prediction block. For example, a video encoder 200 or a video decoder 300 may use such a technique.

[0285] In one example, a subset of the proposed techniques is applied to the AMVP pattern. In one example, a subset of the proposed techniques is applied to the regular merge pattern. In one example, a subset of the proposed techniques is applied to the bilateral matching merge pattern. In one example, a subset of the proposed techniques is applied to the TM merge pattern. In one example, a subset of the proposed techniques is applied to the AMVP merge pattern. In one example, a subset of the proposed techniques is applied to an affine decoded block.

[0286] In one example, the video encoder 200 can signal the LIC index to indicate the selection of a higher-order LIC model. The LIC flag can be extended to accommodate more LIC models. As described above, LIC equal to 0 and 1 correspond to "false" and "true" respectively, while LIC equal to 2 or a large integer value can correspond to a higher-order LIC model, as shown below:

[0287]

[0288] ,

[0289] Where (m, n) {(0, 0)∪S}. S is the coordination set, and when S is empty, the equation degenerates to the same equation as above. For example, S can be, but is not limited to, {(±1,0)}, {(0, ±1)}, {(±1,0), (0, ±1)}, or {(±1,0), (0, ±1), (±1, ±1)}. When the LIC flag > 1, it ensures that the LIC model selection is the same for each merged candidate in the candidate list. Given the motion information of the merged candidate, the model parameters (e.g., α) are derived accordingly. 0,m,n α1,m,n β0, β1 (when needed) are used to minimize the MSE between the current block template and the reference template. For example, a video encoder 200 or a video decoder 300 can use such a technique.

[0290] In another example, the index can also signal this LIC index in AMVP mode, where 0 indicates that the LIC is off, 1 indicates that the traditional LIC is used, and larger integers indicate that a higher-order LIC model is used.

[0291] Simplified decoder-side techniques for merging candidates with inverted LIC flags are now discussed. Decoder-side techniques for motion and prediction refinement (such as DMVR, multi-pass DMVR, BDOF, etc.) can be simplified or omitted to reduce decoding complexity in the average case. The video decoder 300 can utilize such simplified decoder-side techniques. For merging modes, when the LIC flag is inverted (from "true" to "false" or from "false" to "true"), the following examples can be applied, including but not limited to regular merging modes, template matching merging modes, affine merging modes, or sub-block TMVP merging modes.

[0292] In the example, decoder-side motion refinement (DMVR) can be skipped when the LIC flag is flipped.

[0293] In the example, when the LIC flag is flipped, the multiple DMVR passes (described above) can be skipped completely or partially, for example, all passes are skipped, only the first pass is skipped, only the second pass is skipped, only the third pass is skipped, or two out of three passes are skipped.

[0294] In the example, when the LIC flag is flipped, the BDOF used for prediction refinement can be completely skipped. In the example, when the LIC flag is flipped, the multi-iteration MV refinement of the BDOF can be completely skipped.

[0295] In the example, the multi-iteration MV refinement of BDOF can be extended to accommodate more iterations beyond the fourth BDOF MV refinement mentioned above. The BDOF MV refinement can proceed to a maximum of N iterations (where N > 4), and can terminate early before reaching the Nth iteration when deriving zero Vx and Vy during the iteration. When the LIC flag is flipped and the nth (where n < N) BDOF MV refinement derives a non-zero Vx or Vy, the video encoder 200 or video decoder 300 can skip all remaining iterations of the BDOF MV refinement.

[0296] Figure 16This is a flowchart illustrating an example signaling technique, according to one or more aspects of this disclosure, for determining whether to use a first merge list or a second merge list. Video decoder 300 may determine (1300) to decode the current block of video data using a merge mode. For example, video encoder 200 may signal a merge mode flag in the bit stream, the value of which indicates the merge mode to be used for the current block. Video decoder 300 may obtain the merge mode flag from the bit stream and parse the merge mode flag to determine that video decoder 300 should use the merge mode indicated by the merge mode flag to decode the current block.

[0297] The video decoder 300 may obtain a flag (1302) from the bitstream. For example, the video decoder 300 may obtain a flag signaled by the video encoder 200 in the bitstream. This flag may have a value indicating whether a first merge list or a second merge list is used.

[0298] The video decoder 300 may determine a second merge list for the current block based on the value of the flag, wherein the second merge list is based on a first merge list (1304) of the two merge lists. For example, the video decoder 300 may parse the flag to determine the value of the flag, and the value of the flag may instruct the video decoder 300 to use the second merge list.

[0299] The video decoder 300 may decode the current block based on a second merge list (1306). For example, the video decoder 300 may use candidates from the second merge list to decode the current block.

[0300] In some examples, the video decoder 300 may generate a first merge list of two merge lists and generate a second merge list based on the first merge list. In some examples, as part of generating the first merge list, the video decoder 300 may determine multiple merge candidates, each of the multiple merge candidates in the first merge list having a corresponding LIC flag value inherited from its corresponding neighboring block. In some examples, as part of generating the second merge list, the video decoder 300 may copy the multiple merge candidates from the first merge list to the second merge list and change the corresponding LIC flag value of each of the multiple merge candidates to the opposite value. In some examples, the video decoder 300 may trim the multiple merge candidates in the second merge list. In some examples, the video decoder 300 may perform a reordering of the second merge list based on template matching cost.

[0301] In some examples, as part of generating the second merge list, the video decoder 300 may change the LIC flag value associated with the first merge list from "false" to "true" and use the NLIC parameter set of adjacent decoding units to generate two candidates.

[0302] In some examples, as part of generating the second merge list, the video decoder 300 may determine that the value of the LIC flag associated with the first merge list is "true". Based on determining that the value of the LIC flag associated with the first merge list is "true", the video decoder 300 may: a) change the value of the LIC flag associated with the first merge list to "false" for the second merge list; b) keep the value of the LIC flag associated with the first merge list as "true" for the second merge list, and inherit the LIC parameter set from the adjacent CU; or c) keep the value of the LIC flag associated with the first merge list as "true" for the second merge list and inherit the available NLIC parameter set from the adjacent CU.

[0303] In some examples, as part of generating the second merge list, the video decoder 300 may inherit at least one of the LIC parameter set from the adjacent CU and the slope parameter or the offset parameter of the LIC parameter set.

[0304] In some examples, the merging patterns include a regular merging pattern, a bilateral matching merging pattern, a template matching merging pattern, an advanced motion vector prediction sub-merging pattern, or an affine merging pattern. In some examples, the merging pattern includes an affine merging pattern. In some examples, the first merging list includes multiple candidates. In some examples, the multiple candidates include at least one affine merging candidate. In some examples, the multiple candidates further include at least one sub-block temporal motion vector prediction sub-merging candidate.

[0305] In some examples, as part of generating the second merge list, the video decoder 300 may copy multiple candidates from the first merge list to the second merge list, and invert the corresponding LIC flags for each of the multiple candidates in the second merge list. In some examples, the second merge list does not include at least one sub-block temporal motion vector prediction sub-merge candidate. In some examples, as part of generating the second merge list, the video decoder 300 may replace at least one sub-block temporal motion vector prediction sub-merge candidate with at least one other affine merge candidate.

[0306] Figure 17 This is a flowchart illustrating an example technique for template-matching-based local illumination compensation and a second merging list according to one or more aspects of this disclosure. Video decoder 300 can determine (1400) to decode the current block of video data using a merging mode. For example, video encoder 200 can signal a merging mode flag in the bitstream, the value of which indicates the merging mode to be used for the current block. Video decoder 300 can obtain the merging mode flag from the bitstream and parse the merging mode flag to determine that video decoder 300 should use the merging mode indicated by the merging mode flag to decode the current block.

[0307] The video decoder 300 can generate a first merge list for the current block, wherein generating the first merge list includes applying template matching to candidates in the first merge list (1402). For example, the video decoder 300 can use template matching when generating the first merge list.

[0308] The video decoder 300 can generate a second merge list based on the first merge list (1404). For example, the video decoder can use one or more candidates from the first merge list to generate the second merge list.

[0309] The video decoder 300 may use a merging mode and decode the current block based on a first merging list or a second merging list (1406). For example, the video decoder 300 may select a first merging list or a second merging list to select one or more prediction candidates, and may use the selected prediction candidates to decode the current block using a merging mode.

[0310] In some examples, as part of generating the second merge list, the video decoder 300 may change multiple Local Illumination Compensation (LIC) flag values ​​associated with the first merge list to the opposite values ​​for the second merge list before applying template matching to the candidates of the first merge list. In some examples, as part of generating the second merge list, the video decoder 300 may change multiple Local Illumination Compensation (LIC) flag values ​​associated with the first merge list to the opposite values ​​for the second merge list after applying template matching to the candidates of the first merge list.

[0311] In some examples, as part of generating the second merge list, the video decoder 300 may refine the candidates in the first merge list for the second merge list before applying template matching to the candidates in the first merge list. In some examples, as part of generating the second merge list, the video decoder 300 may refine the candidates in the first merge list for the second merge list after applying template matching to the candidates in the first merge list.

[0312] In some examples, as part of generating the second merge list, the video decoder 300 may: determine that one or more candidates in the first merge list are bidirectional predictive merge candidates; and for each bidirectional predictive merge candidate, change the corresponding LIC flag value associated with the first merge list to the opposite LIC flag value for the second merge list. In some examples, as part of generating the second merge list, the video decoder 300 may: determine whether the block size of the current block is greater than a threshold; and based on the determination that the block size of the current block is greater than the threshold, a) determine one or more Local Illumination Compensation (LIC) flag values ​​for the second merge list based on template matching cost, or b) determine one or more LIC flag values ​​for the second merge list by changing the corresponding values ​​of one or more LIC flags in the first merge list for the second merge list.

[0313] In some examples, the video decoder 300 may obtain flags from the bitstream. In some examples, the video decoder 300 may: determine, based on the value of the flags, to use a second merge list to decode the current block; and use the second merge list to decode the current block.

[0314] Figure 18 This is a flowchart illustrating a technique for using local illumination compensation with differential motion vector refinement according to one or more aspects of this disclosure. Video decoder 300 can determine (1500) to decode the current block of video data using a merging mode. For example, video encoder 200 can signal a merging mode flag in the bitstream, the value of which indicates the merging mode to be used for the current block. Video decoder 300 can obtain the merging mode flag from the bitstream and parse the merging mode flag to determine that video decoder 300 should use the merging mode indicated by the merging mode flag to decode the current block.

[0315] The video decoder 300 can determine the application of the LIC (1502) for the current block of video data. For example, the video decoder 300 can determine the application of the LIC based on the LIC flag signaled by the video encoder 200 or by inheriting the LIC parameters from neighboring blocks.

[0316] The video decoder 300 can determine whether to apply DMVR (1504) to the current block. For example, the video decoder 300 may obtain a flag from the bitstream of the video encoder 200, the value of which indicates that the video decoder 300 applies DMVR to the current block and / or determines that the DMVR conditions are met for the current block.

[0317] The video decoder 300 can decode the current block based on both the applied LIC and the applied DMVR (1506). For example, the video decoder 300 can apply both the LIC and the DMVR when decoding the current block.

[0318] In some examples, the video decoder 300 may obtain the LIC flag from the bitstream. In some examples, as part of determining the application of LIC, the video decoder 300 may determine the value of the LIC flag, where the value of the LIC flag indicates the application of LIC.

[0319] In some examples, as part of the application of DMVR, the video decoder 300 may use MRSAD. In some examples, as part of the use of MRSAD, the video decoder 300 may use MRSAD instead of SAD.

[0320] In some examples, the video decoder 300 may apply BDOF to the current block. In some examples, the merging modes include regular merging, bilateral matching merging, template matching merging, advanced motion vector prediction sub-merging, or affine merging. In some examples, the merging mode includes affine merging, and as part of applying DMVR, the video decoder 300 may apply only affine DMVR refinement based on sub-block offsets. In some examples, as part of applying DMVR, the video decoder 300 may apply both affine DMVR refinement based on sub-block offsets and regression-based affine DMVR refinement. In some examples, as part of applying DMVR, the video decoder 300 may apply affine DMVR and skip affine model parameter refinement.

[0321] Figure 19 This is a block diagram illustrating an example video encoder 200 that can perform the techniques of this disclosure. Figure 19 This disclosure is provided for illustrative purposes and should not be construed as a limitation on the techniques extensively illustrated and described herein. For illustrative purposes, this disclosure describes the video encoder 200 in accordance with the techniques of VVC and HEVC. However, the techniques of this disclosure can be performed by video encoding devices configured for other video decoding standards and video decoding formats, such as AV1 and subsequent formats of AV1 video decoding.

[0322] exist Figure 19In the example, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy coding unit 220. Any or all of the video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy coding unit 220 may be implemented in one or more processors or in processing circuitry. For example, the units of the video encoder 200 may be implemented as one or more circuit or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0323] Video data storage 230 is an example of a storage system capable of storing video data to be encoded by components of video encoder 200. Video encoder 200 can receive data from, for example, video source 104 (…). Figure 1 The video data memory 230 receives video data stored in the video data memory 230. The DPB 218 is an example of a memory system that can act as a reference picture memory, storing reference video data for use when the video encoder 200 predicts subsequent video data. The video data memory 230 and DPB 218 can each be formed from any of one or more memory devices or memory cells, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various examples, the video data memory 230 can be on-chip (as illustrated) with other components of the video encoder 200, or off-chip relative to those components.

[0324] In this disclosure, references to video data memory 230 should not be construed as limited to memory inside video encoder 200 (unless specifically described) or memory outside video encoder 200 (unless specifically described). Rather, references to video data memory 230 should be understood as a reference memory that stores video data received by video encoder 200 for encoding (e.g., video data for the current block to be encoded). Figure 1 The memory 106 can also provide temporary storage for the outputs from various units of the video encoder 200.

[0325] Examples Figure 19 Various units help understand the operations performed by the video encoder 200. Units can be implemented as fixed-function circuits, programmable circuits, or combinations thereof. Fixed-function circuits are circuits that provide specific functionality and are pre-defined for the operations that can be performed. Programmable circuits are circuits that can be programmed to perform various tasks and provide flexible functionality for the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units in the unit may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units in the unit may be integrated circuits.

[0326] The video encoder 200 may include an arithmetic logic unit (ALU), an essential function unit (EFU), digital circuitry, analog circuitry, and / or a programmable core, all formed by programmable circuitry. In an example where the operation of the video encoder 200 is performed using software executed by programmable circuitry, memory 106 ( Figure 1 The video encoder 200 may store instructions (e.g., target code) of the software received and executed by the video encoder 200, or another memory (not shown) within the video encoder 200 may store such instructions.

[0327] The video data storage unit 230 is configured to store received video data. The video encoder 200 can retrieve images of the video data from the video data storage unit 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data storage unit 230 can be raw video data to be encoded.

[0328] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-frame prediction unit 226. The mode selection unit 202 may include additional functional units for performing video prediction based on other prediction modes. As an example, the mode selection unit 202 may include a palette unit, an intra-frame block copying unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0329] Mode selection unit 202 typically coordinates multiple coding channels to test combinations of coding parameters and the resulting rate-distortion values ​​for such combinations. Coding parameters may include the CTU-CU partitioning, the prediction mode for the CU, the transformation type of the residual data for the CU, the quantization parameters of the residual data for the CU, etc. Mode selection unit 202 can ultimately select a combination of coding parameters that has a better rate-distortion value compared to other tested combinations.

[0330] The video encoder 200 can divide an image retrieved from the video data storage 230 into a series of CTUs and encapsulate one or more CTUs within a slice. The mode selection unit 202 can divide the image's CTUs according to the tree structure described above (such as an MTT structure, QTBT structure, superblock structure, or the quadtree structure described above). As described above, the video encoder 200 can form one or more CUs by dividing the CTUs according to the tree structure. Such CUs are also commonly referred to as "video blocks" or "blocks".

[0331] Typically, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-frame prediction unit 226) to generate prediction blocks for the current block (e.g., the current CU, or, in HEVC, the overlapping portion of PU and TU). To perform inter-frame prediction for the current block, motion estimation unit 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously decoded pictures stored in DPB 218). Specifically, motion estimation unit 222 may calculate values ​​representing the similarity between a potential reference block and the current block, for example, based on sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc. Motion estimation unit 222 may typically perform these calculations using the sample-by-sample difference between the current block and the reference blocks under consideration. Motion estimation unit 222 may identify the reference block with the lowest value generated by these calculations to indicate the reference block that best matches the current block.

[0332] Motion estimation unit 222 can generate one or more motion vectors (MVs) that define the location of a reference block in a reference image relative to the location of the current block in the current image. Motion estimation unit 222 can then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 can provide a single motion vector, while for bidirectional inter-frame prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vectors to generate a prediction block. For example, motion compensation unit 224 can use the motion vectors to retrieve data for the reference block. As another example, if the motion vectors have fractional sample accuracy, motion compensation unit 224 can interpolate the prediction block according to one or more interpolation filters. Furthermore, for bidirectional inter-frame prediction, motion compensation unit 224 can retrieve data for two reference blocks identified by corresponding motion vectors and combine the retrieved data, for example, by per-sample averaging or weighted averaging.

[0333] When operating according to the AV1 video decoding format, the motion estimation unit 222 and the motion compensation unit 224 can be configured to encode the decoded blocks of video data (e.g., both luma and chroma decoded blocks) using translational motion compensation, affine motion compensation, overlap block motion compensation (OBMC), and / or composite inter-intra-frame prediction.

[0334] As another example, for intra-prediction or intra-prediction decoding, intra-prediction unit 226 may generate a prediction block from samples adjacent to the current block. For example, for directional mode, intra-prediction unit 226 may typically mathematically combine the values ​​of adjacent samples and fill these calculated values ​​across the current block in a defined direction to generate a prediction block. As another example, for DC mode, intra-prediction unit 226 may calculate the average of the adjacent samples of the current block and generate a prediction block to include that resulting average for each sample of the prediction block.

[0335] When operating according to the AV1 video decoding format, the intra-frame prediction unit 226 can be configured to encode decoded blocks of video data (e.g., both luma and chroma decoded blocks) using directional intra-frame prediction, non-directional intra-frame prediction, recursive filter intra-frame prediction, luma-chroma (CFL) prediction, intra-block copying (IBC), and / or palette modes. The mode selection unit 202 may include additional functional units for performing video prediction based on other prediction modes.

[0336] Mode selection unit 202 provides a prediction block to residual generation unit 204. Residual generation unit 204 receives an uncoded raw version of the current block from video data memory 230 and a prediction block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block for the current block. In some examples, residual generation unit 204 may also determine the differences between sample values ​​in the residual block to generate the residual block using residual differential pulse decoding modulation (RDPCM). In some examples, residual generation unit 204 may be formed using one or more subtractor circuits performing binary subtraction.

[0337] In the example where mode selection unit 202 divides a CU into PUs, each PU can be associated with a luma prediction unit and a corresponding chroma prediction unit. Video encoder 200 and video decoder 300 can support PUs of various sizes. As noted above, the size of a CU can refer to the size of the luma decoding block of the CU, while the size of a PU can refer to the size of the luma prediction unit of the PU. Assuming a particular CU size is 2Nx2N, video encoder 200 can support PU sizes of 2Nx2N or NxN for intra-frame prediction, and symmetric PU sizes of 2Nx2N, 2NxN, Nx2N, NxN, or similar for inter-frame prediction. Video encoder 200 and video decoder 300 can also support asymmetric partitioning for PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-frame prediction.

[0338] In an example where mode selection unit 202 does not further divide the CU into PUs, each CU can be associated with a luminance decoding block and a corresponding chrominance decoding block. As mentioned above, the size of the CU can refer to the size of the luminance decoding block of the CU. The video encoder 200 and the video decoder 300 can support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0339] For other video decoding techniques, such as intra-block copy mode decoding, affine mode decoding, and linear model (LM) mode decoding, as some examples, mode selection unit 202 generates a prediction block for the current block being encoded via a corresponding unit associated with the decoding technique. In some examples (such as palette mode decoding), mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how the block is reconstructed based on a selected palette. In such modes, mode selection unit 202 may provide these syntax elements to entropy coding unit 220 for encoding.

[0340] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. Then, the residual generation unit 204 generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0341] Transform processing unit 206 applies one or more transformations to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transformations to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), direction transform, Karhunen-Loeve transform (KLT), or conceptually similar transformations to the residual block. In some examples, transform processing unit 206 may perform multiple transformations on the residual block, such as primary and secondary transformations (e.g., rotation transformations). In some examples, transform processing unit 206 does not apply any transformations to the residual block.

[0342] When operating according to AV1, transform processing unit 206 may apply one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply a combination of horizontal / vertical transforms, which may include the Discrete Cosine Transform (DCT), the Asymmetric Discrete Sine Transform (ADST), the Reversed ADST (e.g., ADST in reverse order), and the Identity Transform (IDTX). When using the Identity Transform, the transform is skipped in either the vertical or horizontal direction. In some examples, transform processing may be skipped entirely.

[0343] Quantization unit 208 quantizes the transform coefficients in a transform coefficient block to generate a quantized transform coefficient block. Quantization unit 208 quantizes the transform coefficients of the transform coefficient block according to the quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may cause information loss, and therefore, the quantized transform coefficients may have lower accuracy compared to the original transform coefficients generated by transform processing unit 206.

[0344] The inverse quantization unit 210 and the inverse transform processing unit 212 can apply inverse quantization and inverse transform, respectively, to the quantized transform coefficient block to reconstruct the residual block based on the transform coefficient block. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block (although potentially with some degree of distortion) based on the reconstructed residual block and the prediction block generated by the mode selection unit 202. For example, the reconstruction unit 214 can add samples of the reconstructed residual block to corresponding samples of the prediction block generated by the mode selection unit 202 to generate the reconstructed block.

[0345] Filter unit 216 may perform one or more filtering operations on the reconstructed block. For example, filter unit 216 may perform a deblocking operation to reduce block artifacts along the edges of the CU. In some examples, the operation of filter unit 216 may be skipped.

[0346] When operating according to AV1, filter unit 216 may perform one or more filtering operations on the reconstructed block. For example, filter unit 216 may perform a deblocking operation to reduce block artifacts along the edges of the CU. In other examples, filter unit 216 may apply a constrained direction enhancement filter (CDEF) after deblocking and may include the application of a non-separable, nonlinear, low-pass directional filter based on the estimated edge direction. Filter unit 216 may also include a loop recovery filter applied after CDEF and may include a separable symmetric normalized Wiener filter or a dual-guided filter.

[0347] The video encoder 200 stores reconstructed blocks in the DPB 218. For example, in an example where the filter unit 216 is not operated, the reconstruction unit 214 may store reconstructed blocks in the DPB 218. In an example where the filter unit 216 is operated, the filter unit 216 may store filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may retrieve reference images formed by the reconstructed (and potentially filtered) blocks from the DPB 218 to perform inter-frame prediction for blocks of subsequent encoded images. Additionally, the intra-frame prediction unit 226 may use the reconstructed blocks of the current image in the DPB 218 to perform intra-frame prediction for other blocks in the current image.

[0348] Typically, entropy coding unit 220 can entropy-encode syntax elements received from other functional components of video encoder 200. For example, entropy coding unit 220 can entropy-encode quantized transform coefficient blocks from quantization unit 208. As another example, entropy coding unit 220 can entropy-encode predictive syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from mode selection unit 202. Entropy coding unit 220 can perform one or more entropy coding operations on syntax elements (another example of video data) to generate entropy-coded data. For example, entropy coding unit 220 can perform context-adaptive variable-length decoding (CAVLC), CABAC, variable-to-variable (V2V) length decoding, syntax-based context-adaptive binary arithmetic decoding (SBAC), probability interval partitioning entropy (PIPE) decoding, exponential Golomb coding, or another type of entropy coding operation on the data. In some examples, entropy coding unit 220 can operate in a bypass mode where syntax elements are not entropy-encoded.

[0349] The video encoder 200 can output a bitstream that includes the entropy coding syntax elements required to reconstruct slices or blocks of images. Specifically, the entropy coding unit 220 can output a bitstream.

[0350] According to AV1, entropy coding unit 220 can be configured as a symbol-to-symbol adaptive multi-symbol arithmetic decoder. The syntax elements in AV1 consist of an N-element alphabet, and the context (e.g., a probability model) consists of a set of N probabilities. Entropy coding unit 220 can store the probabilities as an n-bit (e.g., 15-bit) cumulative distribution function (CDF). Entropy coding unit 220 can perform recursive scaling using an update factor based on the alphabet size to update the context.

[0351] The operations described above are relative to blocks. This description should be understood as operations applied to luma decoding blocks and / or chroma decoding blocks. As described above, in some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the CU. In some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the PU.

[0352] In some examples, it is not necessary to repeat the operations performed relative to the luma decoder for the chroma decoder block. As an example, the operations for identifying the motion vector (MV) and reference image of the luma decoder block do not need to repeat the MV and reference image used to identify the chroma block. Instead, the MV used for the luma decoder block can be scaled to determine the MV used for the chroma block, and the reference image can be the same. As another example, the intra-frame prediction process can be the same for both the luma and chroma decoders.

[0353] Video encoder 200 represents an example of a device configured to encode video data, the device including a memory configured to store video data and one or more processing units implemented in circuitry and configured to perform: determining a merging mode to be applied to the current block of video data; determining whether to apply LIC, independent of whether local illumination compensation (LIC) is applied to adjacent blocks; and encoding the current block based on the determination of the merging mode and whether LIC is applied.

[0354] Figure 20 This is a block diagram illustrating an example video decoder 300 that can perform the techniques of this disclosure. Figure 20 This disclosure is provided for illustrative purposes and not for limiting the techniques extensively illustrated and described herein. For illustrative purposes, the video decoder 300 is described in accordance with VVC and HEVC techniques. However, the techniques of this disclosure can be implemented by video decoding devices configured for other video decoding standards.

[0355] exist Figure 20 In the example, the video decoder 300 includes a decoded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a DPB 314. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 can be implemented in one or more processors or in processing circuitry. For example, the units of the video decoder 300 can be implemented as one or more circuit or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuitry to perform these and other functions.

[0356] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-prediction unit 318. The prediction processing unit 304 may include additional units that perform predictions based on other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an intra-block copying unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0357] When operating according to AV1, motion compensation unit 316 can be configured to decode video data blocks (e.g., both luma and chroma blocks) using translational motion compensation, affine motion compensation, OBMC, and / or composite inter-intra-frame prediction, as described above. Intra-frame prediction unit 318 can be configured to decode video data blocks (e.g., both luma and chroma blocks) using directional intra-frame prediction, non-directional intra-frame prediction, recursive filter intra-frame prediction, CFL, IBC, and / or palette mode, as described above.

[0358] CPB memory 320 is an example of a memory system capable of storing video data (such as encoded video bitstreams) to be decoded by components of video decoder 300. For example, it can be stored from computer-readable medium 110 ( Figure 1 The video data stored in the CPB memory 320 is obtained. The CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. Furthermore, the CPB memory 320 may store video data other than the syntax elements of the decoded image, such as temporary data representing the output from various units of the video decoder 300. The DPB 314 is an example of a memory system that typically stores a decoded image that the video decoder 300 can output, and / or uses as reference video data when decoding subsequent data or images from the encoded video bitstream. The CPB memory 320 and the DPB 314 may each be formed from any of various memory devices or memory cells, such as DRAM (including SDRAM), MRAM, RRAM, or other types of memory devices. The CPB memory 320 and the DPB 314 may be provided by the same memory device or separate memory devices. In various examples, the CPB memory 320 may be on-chip with other components of the video decoder 300, or off-chip relative to those components.

[0359] Additionally or alternatively, in some examples, the video decoder 300 may be from the memory 120 ( Figure 1 The decoded video data can be retrieved from the memory. In other words, memory 120 can utilize CPB memory 320 to store data as discussed above. Similarly, when some or all of the functionality of video decoder 300 is implemented in software to be executed by the processing circuitry of video decoder 300, memory 120 can store instructions to be executed by video decoder 300.

[0360] Examples Figure 20 The various units shown help to understand the operations performed by the video decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Similar to... Figure 19 Fixed-function circuits are circuits that provide specific functionality and are pre-defined for the operations they can perform. Programmable circuits are circuits that can be programmed to perform various tasks and provide flexible functionality for the operations they can perform. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is typically immutable. In some examples, one or more units in a cell may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units in a cell may be integrated circuits.

[0361] The video decoder 300 may include an ALU, EFU, digital circuitry, analog circuitry, and / or a programmable core formed by programmable circuitry. In an example where the operation of the video decoder 300 is performed by software executed on programmable circuitry, on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0362] The entropy decoding unit 302 can receive encoded video data from the CPB and perform entropy decoding on the video data to reproduce the syntax elements. The prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, and the filter unit 312 can generate decoded video data based on the syntax elements extracted from the bitstream.

[0363] Typically, the video decoder 300 reconstructs the image block by block. The video decoder 300 may perform reconstruction operations on each block individually (where the block currently being reconstructed (e.g., being decoded) may be referred to as the "current block").

[0364] Entropy decoding unit 302 can entropy decode the syntax elements of the quantized transform coefficients defining the quantized transform coefficient block, as well as transform information (such as quantization parameters (QP) and / or transform mode indications). Inverse quantization unit 306 can use the QP associated with the quantized transform coefficient block to determine the degree of quantization, and similarly, determine the degree of inverse quantization to be applied by inverse quantization unit 306. Inverse quantization unit 306 can, for example, perform a bit-by-bit left shift operation to inverse quantize the quantized transform coefficients. Inverse quantization unit 306 can thereby form a transform coefficient block including the transform coefficients.

[0365] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 may apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotation transform, an inverse direction transform, or another inverse transform to the transform coefficient block.

[0366] Furthermore, the prediction processing unit 304 generates a prediction block based on the prediction information syntax elements entropy decoded by the entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is an inter-frame prediction, the motion compensation unit 316 can generate the prediction block. In this case, the prediction information syntax elements may indicate the reference picture from which the reference block is to be retrieved in the DPB 314, and a motion vector identifying the position of the reference block in the reference picture relative to the position of the current block in the current picture. The motion compensation unit 316 may generally follow the same procedure as relative to the motion compensation unit 224 ( Figure 19 The method described is essentially the same as the method used to perform the inter-frame prediction process.

[0367] As another example, when the prediction information syntax element indicates that the current block is intra-predictive, intra-predictive unit 318 may generate a prediction block according to the intra-predictive mode indicated by the prediction information syntax element. Similarly, intra-predictive unit 318 may generally follow the same procedure as relative to intra-predictive unit 226 ( Figure 19 The intra-prediction process is performed in a manner substantially similar to that described above. The intra-prediction unit 318 can retrieve data of neighboring samples of the current block from the DPB 314.

[0368] Reconstruction unit 310 can use the prediction block and the residual block to reconstruct the current block. For example, reconstruction unit 310 can add samples from the residual block to the corresponding samples from the prediction block to reconstruct the current block.

[0369] Filter unit 312 can perform one or more filtering operations on the reconstructed block. For example, filter unit 312 can perform a deblocking operation to reduce block artifacts along the edges of the reconstructed block. The operation of filter unit 312 is not necessarily performed in all examples.

[0370] The video decoder 300 can store reconstructed blocks in the DPB 314. For example, in an example where the operation of the filter unit 312 is not performed, the reconstruction unit 310 can store the reconstructed blocks in the DPB 314. In an example where the operation of the filter unit 312 is performed, the filter unit 312 can store the filtered reconstructed blocks in the DPB 314. As discussed above, the DPB 314 can provide reference information (such as samples of the current image for intra-frame prediction and samples of previously decoded images for subsequent motion compensation) to the prediction processing unit 304. Furthermore, the video decoder 300 can output decoded images (e.g., decoded video) from the DPB 314 for use in applications such as... Figure 1 The subsequent presentation on display devices such as display device 118.

[0371] In this manner, video decoder 300 represents an example of a video decoding device, which includes a memory configured to store video data and one or more processing units implemented in circuitry and configured to perform: determining a merging mode to be applied to the current block of video data; determining whether to apply LIC, independent of whether local illumination compensation (LIC) is applied to adjacent blocks; and decoding the current block based on the determination of the merging mode and whether LIC is applied.

[0372] Figure 21 This is an example technical flowchart illustrating the technology for encoding the current block according to the present disclosure. The current block may be or may include the current CU. Although relative to the video encoder 200 ( Figure 1 and Figure 16 This is a description, but it should be understood that other devices can be configured to perform similar actions. Figure 21 Similar to the technology.

[0373] In this example, the video encoder 200 initially predicts the current block (400). For example, the video encoder 200 may form a prediction block for the current block. The video encoder 200 may then compute a residual block for the current block (402). To compute the residual block, the video encoder 200 may compute the difference between the unencoded original block for the current block and the prediction block. The video encoder 200 may then transform the residual block and quantize the transform coefficients of the residual block (404). Next, the video encoder 200 may scan the quantized transform coefficients of the residual block (406). During or after the scan, the video encoder 200 may entropy encode the transform coefficients (408). For example, the video encoder 200 may use CAVLC or CABAC to encode the transform coefficients. The video encoder 200 may then output the entropy-encoded data of the block (410).

[0374] Figure 22This is a flowchart illustrating an example technique for decoding a current block of video data according to the present disclosure. The current block may be or may include the current CU. Although relative to video decoder 300 ( Figure 1 and Figure 20 This is a description, but it should be understood that other devices can be configured to perform similar actions. Figure 22 Similar to the technology.

[0375] The video decoder 300 may receive entropy-coded data for the current block, such as entropy-coded prediction information and entropy-coded data for the transform coefficients of the residual block corresponding to the current block (500). The video decoder 300 may entropy decode the entropy-coded data to determine prediction information for the current block and reproduce the transform coefficients of the residual block (502). The video decoder 300 may, for example, use an intra-frame prediction mode or inter-frame prediction mode indicated by the prediction information of the current block to predict the current block (504) to compute a prediction block for the current block. The video decoder 300 may then perform an inverse scan on the reproduced transform coefficients (506) to create a block of quantized transform coefficients. The video decoder 300 may then inverse quantize the transform coefficients and apply an inverse transform to the transform coefficients to produce a residual block (508). The video decoder 300 may finally decode the current block by combining the prediction block and the residual block (510).

[0376] The following numbered clauses illustrate one or more aspects of the devices and technologies described in this disclosure.

[0377] Clause 1A. A method for decoding video data, the method comprising: determining a merging mode to be applied to a current block of the video data; determining whether to apply LIC, independent of whether local illumination compensation (LIC) is applied to adjacent blocks; and decoding the current block based on the merging mode and the determination of whether LIC is applied.

[0378] Clause 2A. The method according to Clause 1A, wherein determining whether to apply LIC includes disabling LIC for the current block based on the determination of applying a merge mode to the current block.

[0379] Clause 3A. The method according to Clause 1A, wherein determining whether to apply LIC includes enabling LIC for the current block based on the determination of applying a merge mode to the current block.

[0380] Clause 4A. The method according to Clause 1A, the method further comprising signaling or resolving an indication of whether to apply the LIC to the LIC flag of the current block.

[0381] Clause 5A. The method according to any one of Clauses 1A to 4A, the method further comprising reordering the candidate list before applying the LIC model.

[0382] Clause 6A. The method according to any one of Clauses 1A to 5A, the method further comprising: determining whether one or more decoder-side motion vector refinement (DMVR) conditions are met; and taking action based on whether one or more DMVR conditions are met.

[0383] Clause 7A. The action described in accordance with Clause 6A includes one of the following: applying DMVR and LIC; applying DMVR without applying LIC; or applying DMVR without applying LIC and setting the LIC flag to "false".

[0384] Clause 8A. The method according to any one of Clauses 1A to 7A, the method further comprising: generating a first merge list; and generating a second merge list, wherein at least one of the first merge list or the second merge list is used to determine whether to apply the LIC.

[0385] Clause 9A. The method described in Clause 8A, the method further comprising combining the first merge list and the second merge list.

[0386] Clause 10A. The method described in Clause 9A, wherein the combination includes cascading or interleaving.

[0387] Clause 11A. The method according to any one of Clauses 1A to 10A, the method further comprising: generating a new merge list for merging mode LIC flag signaling; and determining, for the new merge list, bidirectional prediction (BCW) values ​​with CU level weights.

[0388] Clause 12A. The method according to Clause 11A, wherein determining the BCW value includes setting the BCW value to a preset value or independently deriving the BCW value from the original merge list.

[0389] Clause 13A. The method according to any one of Clauses 1A, or 3A to 10A, further comprising: determining that LIC is enabled for the current block; and using LIC to derive parameters for weighted prediction of the current block.

[0390] Clause 14A. The method according to any one of Clauses 1A, or 3A to 11A, the method further comprising: determining to apply LIC to the current block; determining whether to subsample a template to derive a LIC model for the current block; and, based on the determination not to subsample the template, including additional samples to derive the LIC model.

[0391] Clause 15A. The method according to any one of Clauses 1A, or 3A to 14A, the method further comprising: determining to apply LIC to the current block; determining whether one or more bidirectional optical flow (BDOF) conditions are satisfied; and applying BDOF to the current block based on the determination of applying LIC and the determination of satisfying one or more BDOF conditions.

[0392] Clause 16A. The method according to Clause 15A, wherein the one or more BDOF conditions include: a) the current block is a bidirectional predictive decoded block; b) the current block has a first reference block in a first reference picture and a second reference block in a second reference picture, the first reference picture being a past reference picture compared to the current picture, and the second reference picture being a future reference picture compared to the current picture; c) the picture order count (POC) distance between the current picture and the first reference picture is equal to the POC distance between the current picture and the second reference picture; or d) the current block has equal weights for each reference block.

[0393] Clause 17A. The method according to any one of Clauses 1A, or 3A to 16A, further comprising: determining to apply LIC to the current block; determining whether one or more bilateral decoder-side motion vector refinement (BDMVR) conditions are satisfied; and applying BDMVR to the current block based on the determination of applying LIC and the determination of satisfying one or more BDMVR conditions.

[0394] Clause 18A. The method according to Clause 17A, wherein the one or more BDMVR conditions include: a) the current block is a bidirectional predictive decoded block; b) the current block has a first reference block in a first reference picture and a second reference block in a second reference picture, the first reference picture being a past reference picture compared to the current picture, and the second reference picture being a future reference picture compared to the current picture; or c) the picture sequence count (POC) distance between the current picture and the first reference picture is equal to the POC distance between the current picture and the second reference picture;

[0395] Clause 19A. The method according to any one of Clauses 4A to 18A, wherein the value of the LIC flag, which is equal to or greater than 1, indicates the application of a higher-order LIC model.

[0396] Clause 20A. The method according to any one of Clauses 1A to 19A, wherein decoding includes decoding.

[0397] Clause 21A. The method according to any one of Clauses 1A to 20A, wherein decoding includes encoding.

[0398] Clause 22A. An apparatus for decoding video data, the apparatus comprising one or more components for performing the method according to any one of Clauses 1A to 19A.

[0399] Clause 23A. The device according to Clause 22A, wherein the one or more components include one or more processors implemented in a circuit.

[0400] Clause 24A. The device according to any one of Clauses 22A and 23A, the device further includes one or more memories for storing the video data.

[0401] Clause 25A. The device according to any one of Clauses 22A to 24A, the device further comprising a display configured to display decoded video data.

[0402] Clause 26A. The device pursuant to any one of Clauses 22A to 25A, wherein the device includes one or more of a camera, computer, mobile device, broadcast receiver device or set-top box.

[0403] Clause 27A. The device pursuant to any one of Clauses 22A to 26A, wherein the device includes a video decoder.

[0404] Clause 28A. The device pursuant to any one of Clauses 22A to 27A, wherein the device includes a video encoder.

[0405] Clause 29A. A computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform the method according to any one of Clauses 1A to 19A.

[0406] Clause 1B. A method for decoding video data, the method comprising: determining to decode a current block of the video data using a merging mode; obtaining a flag from a bitstream; determining, based on the value of the flag, a second merging list to use for the current block from two merging lists, wherein the second merging list is based on a first merging list from the two merging lists; and decoding the current block based on the second merging list.

[0407] Clause 2B. The method according to Clause 1B, the method further comprising: generating a first merge list of the two merge lists; and generating a second merge list based on the first merge list.

[0408] Clause 3B. The method according to Clause 2B, wherein generating the first merge list includes determining a plurality of merge candidates, each of the plurality of merge candidates in the first merge list having a corresponding local illumination compensation (LIC) flag value inherited from a corresponding adjacent block, and wherein generating the second merge list includes copying the plurality of merge candidates from the first merge list to the second merge list and changing the corresponding LIC flag value of each of the plurality of merge candidates to the opposite value.

[0409] Clause 4B. The method according to Clause 3B, the method further comprising trimming the plurality of merge candidates in the second merge list.

[0410] Clause 5B. The method described in Clause 4B, the method further comprising performing a reordering of the second merged list based on template matching cost.

[0411] Clause 6B. The method according to any one of Clauses 2B to 5B, wherein generating the second merge list includes changing the local illumination compensation (LIC) flag value associated with the first merge list from "false" to "true", and using the non-local illumination compensation (NLIC) parameter set of adjacent decoding units to generate two candidates.

[0412] Clause 7B. The method according to any one of Clauses 2B to 5B, wherein generating the second merge list comprises: determining that the value of a Local Illumination Compensation (LIC) flag associated with the first merge list is “true”; and based on the determination that the value of the LIC flag associated with the first merge list is “true”, a) changing the value of the LIC flag associated with the first merge list to “false” for use in the second merge list, b) keeping the value of the LIC flag associated with the first merge list as “true” for use in the second merge list and inheriting the LIC parameter set from the adjacent CU, or c) keeping the value of the LIC flag associated with the first merge list as “true” for use in the second merge list and inheriting the available Non-Local Illumination Compensation (NLIC) parameter set from the adjacent CU.

[0413] Clause 8B. The method according to any one of Clauses 2B to 7B, wherein generating the second merge list comprises: inheriting a set of local illumination compensation (LIC) parameters from an adjacent CU; and adjusting at least one of a slope parameter or an offset parameter of the LIC parameter set.

[0414] Clause 9B. The method according to any one of Clauses 1B to 8B, wherein the merging mode includes a regular merging mode, a bilateral matching merging mode, a template matching merging mode, an advanced motion vector predictor sub-merging mode, or an affine merging mode.

[0415] Clause 10B. The method according to Clause 9B, wherein the merging mode includes the affine merging mode, and wherein the first merging list includes a plurality of candidates, wherein the plurality of candidates includes at least one affine merging candidate and at least one sub-block temporal motion vector prediction sub-merging candidate.

[0416] Clause 11B. The method according to Clause 10B, the method further comprising generating a second merge list based on the first merge list, wherein generating the second merge list comprises copying the plurality of candidates from the first merge list to the second merge list and inverting the corresponding local illumination compensation (LIC) flags for the respective candidates among the plurality of candidates in the second merge list, and wherein the second merge list does not include the at least one sub-block temporal motion vector prediction sub-merge candidate.

[0417] Clause 12B. The method according to Clause 11B, wherein generating the second merge list further comprises replacing the at least one sub-block time motion vector prediction sub-merge candidate with at least one other affine merge candidate.

[0418] Clause 13B. An apparatus for decoding video data, the apparatus comprising: one or more memories for storing the video data; and one or more processors implemented in circuitry and communicatively coupled to the one or more memories, the one or more processors being configured to: determine a current block of the video data to be decoded using a merging mode; obtain a flag from a bitstream; determine, based on the value of the flag, a second merging list to be used for the current block from two merging lists, wherein the second merging list is based on a first merging list from the two merging lists; and decode the current block based on the second merging list.

[0419] Clause 14B. The device according to Clause 13B, wherein the one or more processors are further configured to: generate a first merge list from the two merge lists; and generate a second merge list based on the first merge list.

[0420] Clause 15B. The apparatus according to Clause 14B, wherein, as part of generating the first merge list, the one or more processors are configured to determine a plurality of merge candidates, each of the plurality of merge candidates in the first merge list having a corresponding Local Illumination Compensation (LIC) flag value inherited from a corresponding adjacent block, and wherein, as part of generating the second merge list, the one or more processors are configured to copy the plurality of merge candidates from the first merge list to the second merge list and change the corresponding LIC flag value of each of the plurality of merge candidates to an opposite value.

[0421] Clause 16B. The device as described in Clause 15B, wherein one or more processors are further configured to prune the plurality of merge candidates in the second merge list.

[0422] Clause 17B. The apparatus according to any one of Clauses 14B to 16B, wherein, as part of generating the second merge list, the one or more processors are configured to change the local illumination compensation (LIC) flag value associated with the first merge list from “false” to “true”, and to generate two candidates using the non-local illumination compensation (NLIC) parameter set of adjacent decoding units.

[0423] Clause 18B. The device according to any one of Clauses 14B to 17B, wherein, as part of generating the second merge list, the one or more processors are configured to: inherit a set of local illumination compensation (LIC) parameters from an adjacent CU; and adjust at least one of a slope parameter or an offset parameter of the LIC parameter set.

[0424] Clause 19B. The device pursuant to any one of Clauses 13B to 18B, wherein the merging mode includes a regular merging mode, a bilateral matching merging mode, a template matching merging mode, an advanced motion vector prediction sub-merging mode, or an affine merging mode.

[0425] Clause 20B. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: determine to decode a current block of video data using a merging mode; obtain a flag from a bitstream; determine, based on the value of the flag, to use a second merging list from two merging lists for the current block, wherein the second merging list is based on a first merging list from the two merging lists; and decode the current block based on the second merging list.

[0426] Clause 1C. A method for decoding video data, the method comprising: determining a current block of the video data to be decoded using a merging mode; generating a first merging list of the current block, wherein generating the first merging list includes applying template matching to candidates in the first merging list; generating a second merging list based on the first merging list; and decoding the current block using the merging mode and based on either the first merging list or the second merging list.

[0427] Clause 2C. The method according to Clause 1C, wherein generating the second merge list comprises: changing a plurality of local illumination compensation (LIC) flag values ​​associated with the first merge list to opposite values ​​for the second merge list before applying template matching to the candidates of the first merge list.

[0428] Clause 3C. The method according to Clause 1C, wherein generating the second merge list comprises: after applying template matching to the candidates of the first merge list, changing a plurality of local illumination compensation (LIC) flag values ​​associated with the first merge list to opposite values ​​for the second merge list.

[0429] Clause 4C. The method according to any one of Clauses 1C to 3C, wherein generating the second merge list comprises: trimming the candidates of the first merge list for the second merge list before applying template matching to the candidates of the first merge list.

[0430] Clause 5C. The method according to any one of Clauses 1C to 3C, wherein generating the second merge list comprises: after applying template matching to the candidates of the first merge list, trimming the candidates of the first merge list for the second merge list.

[0431] Clause 6C. The method according to any one of Clauses 1C to 5C, wherein generating the second merge list comprises: determining that one or more candidates in the first merge list are bidirectional predictive merge candidates; and for each bidirectional predictive merge candidate in the bidirectional predictive merge list, changing the corresponding local illumination compensation (LIC) flag value associated with the first merge list to the opposite LIC flag value for the second merge list.

[0432] Clause 7C. The method according to any one of Clauses 1C to 6C, wherein generating the second merge list comprises: determining whether the block size of the current block is greater than a threshold; and based on the determination that the block size of the current block is greater than the threshold, a) determining one or more Local Illumination Compensation (LIC) flag values ​​for the second merge list based on template matching cost, or b) determining the one or more LIC flag values ​​for the second merge list by changing the corresponding values ​​of one or more LIC flags of the first merge list for the second merge list.

[0433] Clause 8C. The method according to any one of Clauses 1C to 7C, the method further comprising: obtaining a flag from a bitstream; determining, based on the value of the flag, to decode the current block using the second merge list; and decoding the current block using the second merge list.

[0434] Clause 9C. An apparatus for decoding video data, the apparatus comprising: one or more memories for storing the video data; and one or more processors implemented in circuitry and communicatively coupled to the one or more memories, the one or more processors being configured to: determine a current block of the video data to be decoded using a merging mode; generate a first merging list of the current block, including applying template matching to candidates in the first merging list; generate a second merging list based on the first merging list; and decode the current block using the merging mode and based on either the first merging list or the second merging list.

[0435] Clause 10C. The apparatus according to Clause 9C, wherein, as part of generating the second merge list, the one or more processors are configured to: change a plurality of local illumination compensation (LIC) flag values ​​associated with the first merge list to opposite values ​​for the second merge list before applying template matching to the candidates of the first merge list.

[0436] Clause 11C. The apparatus according to Clause 9C, wherein, as part of generating the second merge list, the one or more processors are configured to: after applying template matching to the candidates of the first merge list, change a plurality of local illumination compensation (LIC) flag values ​​associated with the first merge list to opposite values ​​for the second merge list.

[0437] Clause 12C. The device pursuant to any one of Clauses 9C to 11C, wherein, as part of generating the second merge list, the one or more processors are configured to trim the candidates of the first merge list for the second merge list before applying template matching to the candidates of the first merge list.

[0438] Clause 13C. The device pursuant to any one of Clauses 9C to 11C, wherein, as part of generating the second merge list, the one or more processors are configured to prune the candidates of the first merge list for the second merge list after applying template matching to the candidates of the first merge list.

[0439] Clause 14C. The device according to any one of Clauses 9C to 13C, wherein, as part of generating the second merge list, the one or more processors are configured to: determine that one or more candidates in the first merge list are bidirectional predictive merge candidates; and for each of the bidirectional predictive merge candidates, change the corresponding Local Illumination Compensation (LIC) flag value associated with the first merge list to the opposite LIC flag value for the second merge list.

[0440] Clause 15C. The device according to any one of Clauses 9C to 15C, wherein, as part of generating the second merge list, the one or more processors are configured to: determine whether the block size of the current block is greater than a threshold; and based on the determination that the block size of the current block is greater than the threshold, a) determine one or more Local Illumination Compensation (LIC) flag values ​​for the second merge list based on template matching cost, or b) determine the one or more LIC flag values ​​for the second merge list by changing the corresponding values ​​of one or more LIC flags of the first merge list for the second merge list.

[0441] Clause 16C. The apparatus of any one of Clauses 9C to 15C, wherein the one or more processors are further configured to: obtain a flag from a bitstream; determine, based on the value of the flag, to use the second merge list to decode the current block; and use the second merge list to decode the current block.

[0442] Clause 17C. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: determine a current block of video data to be decoded using a merging mode; generate a first merging list of the current block, including applying template matching to candidates in the first merging list; generate a second merging list based on the first merging list; and decode the current block using the merging mode and based on either the first merging list or the second merging list.

[0443] Clause 18C. A non-transitory computer-readable storage medium as described in Clause 17C, wherein, as part of generating the second merge list, the instructions cause the one or more processors to: change a plurality of local illumination compensation (LIC) flag values ​​associated with the first merge list to opposite values ​​for the second merge list before applying template matching to the candidates of the first merge list.

[0444] Clause 19C. A non-transitory computer-readable storage medium as described in Clause 17C, wherein, as part of generating the second merge list, the instructions cause the one or more processors to: after applying template matching to the candidates of the first merge list, change a plurality of local illumination compensation (LIC) flag values ​​associated with the first merge list to opposite values ​​for the second merge list.

[0445] Clause 20C. A nontransitory computer-readable storage medium according to any one of Clauses 17C to 19C, wherein the instructions further cause the one or more processors to: obtain a flag from a bit stream; determine, based on the value of the flag, to use the second merge list to decode the current block; and use the second merge list to decode the current block.

[0446] Clause 1D. A method for decoding video data, the method comprising: determining to decode a current block of the video data using a merging mode; determining to apply Local Illumination Compensation (LIC) to the current block; determining to apply Decoder-Side Motion Vector Refinement (DMVR) to the current block; and decoding the current block based on the application of LIC and the application of DMVR.

[0447] Clause 2D. The method according to Clause 1D, the method further comprising obtaining a LIC flag in a bit stream, wherein determining the application of a LIC comprises determining a value of the LIC flag, and wherein the value of the LIC flag indicates the application of a LIC.

[0448] Clause 3D. The method according to any one of Clauses 1D to 2D, wherein applying DMVR includes using the mean to remove the sum of absolute differences (MRSAD).

[0449] Clause 4D. The method described in Clause 3D, wherein the use of MRSAD includes using MRSAD instead of SAD.

[0450] Clause 5D. The method according to any one of Clauses 1D to 4D, the method further comprising applying bidirectional optical flow (BDOF) to the current block.

[0451] Clause 6D. The method according to any one of Clauses 1D to 5D, wherein determining the application of DMVR includes determining that one or more DMVR conditions are met.

[0452] Clause 7D. The method according to any one of Clauses 1D to 6D, wherein the merging mode includes a regular merging mode, a bilateral matching merging mode, a template matching merging mode, an advanced motion vector prediction sub-merging mode, or an affine merging mode.

[0453] Clause 8D. The method according to Clause 7D, wherein the merging mode includes the affine merging mode, and wherein applying DMVR includes applying only affine DMVR refinement based on sub-block offset.

[0454] Clause 9D. The method according to Clause 7D, wherein the merging mode includes the affine merging mode, and wherein applying DMVR includes applying sub-block offset-based affine DMVR refinement and regression-based affine DMVR refinement.

[0455] Clause 10D. The method according to Clause 7D, wherein the merging mode includes the affine merging mode, and wherein applying DMVR includes applying affine DMVR and bypassing affine model parameter refinement.

[0456] Clause 11D. An apparatus for decoding video data, the apparatus comprising: one or more memories for storing the video data; and one or more processors implemented in circuitry and communicatively coupled to the one or more memories, the one or more processors being configured to: determine to decode a current block of the video data using a merging mode; determine to apply Local Illumination Compensation (LIC) to the current block; determine to apply Decoder-Side Motion Vector Refinement (DMVR) to the current block; and decode the current block based on the application of LIC and the application of DMVR.

[0457] Clause 12D. The apparatus of Clause 11D, wherein the one or more processors are further configured to obtain an LIC flag in a bit stream, wherein, as part of determining an applied LIC, the one or more processors are configured to determine a value of the LIC flag, and wherein the value of the LIC flag indicates an applied LIC.

[0458] Clause 13D. A device pursuant to any one of Clauses 12D to 13D, wherein, as part of applying DMVR, the one or more processors are configured to use mean-removed sum of absolute differences (MRSAD).

[0459] Clause 14D. The device described in Clause 13D, wherein the use of MRSAD includes the use of MRSAD instead of SAD.

[0460] Clause 15D. The device according to any one of Clauses 11D to 14D, wherein the one or more processors are further configured to apply bidirectional optical flow (BDOF) to the current block.

[0461] Clause 16D. The device pursuant to any one of Clauses 11D to 15D, wherein, as part of determining the application of DMVR, the one or more processors are configured to determine that one or more DMVR conditions are satisfied.

[0462] Clause 17D. The device according to any one of Clauses 11D to 16D, wherein the merging mode includes a regular merging mode, a bilateral matching merging mode, a template matching merging mode, an advanced motion vector predictor sub-merging mode, or an affine merging mode.

[0463] Clause 18D. The device according to Clause 17D, wherein the merging mode includes the affine merging mode, and wherein, as part of applying DMVR, the one or more processors are configured to apply only affine DMVR refinement based on sub-block offset.

[0464] Clause 19D. The device according to Clause 17D, wherein the merging mode includes the affine merging mode, and wherein, as part of applying DMVR, the one or more processors are configured to apply sub-block offset-based affine DMVR refinement and regression-based affine DMVR refinement.

[0465] Clause 20D. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: determine to decode a current block of video data using a merging mode; determine to apply Local Illumination Compensation (LIC) to the current block; determine to apply Decoder-Side Motion Vector Refinement (DMVR) to the current block; and decode the current block based on the application of LIC and the application of DMVR.

[0466] It should be recognized that, based on the examples, certain actions or events of any technique described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary for implementing the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0467] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0468] By way of example, and not limitation, such computer-readable storage media may include one or more of RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (such as infrared, radio, and microwave) is included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. Combinations of these should also be included within the scope of computer-readable media.

[0469] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0470] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units may be combined in a codec hardware unit, or various units may be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0471] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for decoding video data, the method comprising: Determine to use the merge mode to decode the current block of the video data; Determine to apply Local Illumination Compensation (LIC) to the current block; Determine whether to apply decoder-side motion vector refinement (DMVR) to the current block. as well as The current block is decoded based on the application LIC and the application DMVR.

2. The method of claim 1, further comprising obtaining an LIC flag in a bitstream, wherein determining the application of an LIC includes determining a value of the LIC flag, and wherein the value of the LIC flag indicates the application of an LIC.

3. The method of claim 1, wherein applying DMVR includes removing the sum of absolute differences (MRSAD) using the mean.

4. The method of claim 3, wherein using MRSAD includes using MRSAD instead of SAD.

5. The method of claim 1, further comprising applying bidirectional optical flow (BDOF) to the current block.

6. The method of claim 1, wherein determining the application of DMVR includes determining that one or more DMVR conditions are met.

7. The method according to claim 1, wherein the merging mode includes a regular merging mode, a bilateral matching merging mode, a template matching merging mode, an advanced motion vector prediction sub-merging mode, or an affine merging mode.

8. The method of claim 7, wherein the merging mode includes the affine merging mode, and wherein applying DMVR includes applying only affine DMVR refinement based on sub-block offset.

9. The method of claim 7, wherein the merging mode includes the affine merging mode, and wherein applying DMVR includes applying sub-block offset-based affine DMVR refinement and regression-based affine DMVR refinement.

10. The method of claim 7, wherein the merging mode includes the affine merging mode, and wherein applying DMVR includes applying affine DMVR and bypassing affine model parameter refinement.

11. An apparatus for decoding video data, the apparatus comprising: One or more memories, the one or more memories being used to store the video data; and One or more processors, implemented in a circuit and communicatively coupled to the one or more memories, are configured to: Determine to use the merge mode to decode the current block of the video data; Determine to apply Local Illumination Compensation (LIC) to the current block; Determine whether to apply decoder-side motion vector refinement (DMVR) to the current block. as well as The current block is decoded based on the application LIC and the application DMVR.

12. The apparatus of claim 11, wherein the one or more processors are further configured to obtain an LIC flag in a bitstream, wherein, as part of determining an applied LIC, the one or more processors are configured to determine a value of the LIC flag, and wherein the value of the LIC flag indicates an applied LIC.

13. The device of claim 12, wherein, as part of applying DMVR, the one or more processors are configured to use mean-removed sum of absolute differences (MRSAD).

14. The device of claim 13, wherein using MRSAD includes using MRSAD instead of SAD.

15. The device of claim 11, wherein the one or more processors are further configured to apply bidirectional optical flow (BDOF) to the current block.

16. The device of claim 11, wherein, as part of determining the application of DMVR, the one or more processors are configured to determine that one or more DMVR conditions are satisfied.

17. The device of claim 11, wherein the merging mode includes a conventional merging mode, a bilateral matching merging mode, a template matching merging mode, an advanced motion vector prediction sub-merging mode, or an affine merging mode.

18. The device of claim 17, wherein the merging mode includes the affine merging mode, and wherein, as part of applying DMVR, the one or more processors are configured to apply only affine DMVR refinement based on sub-block offset.

19. The apparatus of claim 17, wherein the merging mode includes the affine merging mode, and wherein, as part of applying DMVR, the one or more processors are configured to apply sub-block offset-based affine DMVR refinement and regression-based affine DMVR refinement.

20. A non-transitory computer-readable storage medium storing instructions, said instructions, when executed, causing one or more processors to: Determine to use the merge mode to decode the current block of video data; Determine to apply Local Illumination Compensation (LIC) to the current block; Determine whether to apply decoder-side motion vector refinement (DMVR) to the current block. as well as The current block is decoded based on the application LIC and the application DMVR.

Images