Local illumination compensation

Abstract

A method of motion improvement based on optical flow is provided. An exemplary method includes receiving a bitstream; and decoding one or more images using the encoded information of the bitstream, the decoding including: determining whether a current block is encoded by being divided into a plurality of portions; and when the current block is divided into a plurality of portions for inter prediction according to corresponding reference blocks, determining whether to apply a local illumination compensation (LIC) process to the plurality of portions based on whether an encoded block has applied the LIC process.

Description

[0001] Cross-references to related applications This disclosure claims priority to U.S. Provisional Application No. 63 / 587,169, filed October 2, 2023, and also claims priority to U.S. Patent Application No. 18 / 895,032, filed September 24, 2024, entitled "Local Illumination Compensation." Both applications are incorporated herein by reference in their entirety. Technical Field

[0002] This disclosure relates generally to video processing, and more specifically, to methods for local illumination compensation. Background Technology

[0003] Video consists of a set of still images (or "frames") that capture visual information. To reduce storage memory and transmission bandwidth, video can be compressed before storage or transmission and decompressed before display. The compression process is usually called encoding, while the decompression process is usually called decoding. There are many video coding formats that use standardized video coding techniques, the most common being based on prediction, transform, quantization, entropy coding, and loop filtering. Standardization organizations have developed video coding standards that specify particular video coding formats, such as the High Efficiency Video Coding (HEVC / H.265) standard, the Versatile Video Coding (VVC / H.266) standard, and the AVS standard. As video standards adopt increasingly advanced video coding techniques, the coding efficiency of new video coding standards also increases. Summary of the Invention

[0004] Embodiments of this disclosure provide methods for local illumination compensation. Specifically, embodiments of this disclosure provide a method for decoding a bitstream to output one or more images of a video stream, a method for encoding a video sequence into a bitstream, and a non-transitory computer-readable storage medium for storing the video bitstream.

[0005] In a first aspect, according to some exemplary embodiments, a method is provided for decoding a bitstream to output one or more images in a video stream, the method comprising: receiving a bitstream; and using encoded information in the bitstream to decode one or more images, wherein the decoding comprises: determining whether a current block of a current image is encoded by dividing it into multiple parts; and when the current block is divided into multiple parts, determining whether to apply the LIC process to the multiple parts based on whether a local illumination compensation (LIC) process is applied to the encoded blocks.

[0006] In a second aspect, according to some exemplary embodiments, a method for encoding a video sequence into a bitstream is provided, the method comprising: receiving a video sequence; encoding one or more images of the video sequence; and generating a bitstream, wherein the encoding comprises: determining whether a current block of a current image is encoded by dividing it into multiple parts; and when the current block is divided into multiple parts for inter-frame prediction based on corresponding reference blocks, determining whether to apply the LIC process to the multiple parts based on whether a Local Illumination Compensation (LIC) process is applied to the encoded blocks.

[0007] In a third aspect, according to some exemplary embodiments, a decoding apparatus is provided, the decoding apparatus comprising: a receiving module configured to receive a bitstream; and a decoding module configured to decode one or more images using encoded information in the bitstream, wherein the decoding module is configured to: determine whether a current block of a current image is encoded by dividing it into multiple parts; and when the current block is divided into multiple parts, determine whether to apply the LIC process to the multiple parts based on whether a Local Illumination Compensation (LIC) process is applied to the encoded blocks.

[0008] In a fourth aspect, according to some exemplary embodiments, an encoding apparatus is provided, the encoding apparatus comprising: a receiving module configured to receive a video sequence; an encoding module configured to encode one or more images in the video sequence; and a generating module configured to generate a bitstream, wherein the encoding module is configured to: determine whether a current block of a current image is encoded by dividing it into multiple parts; and when the current block is divided into multiple parts for inter-frame prediction based on corresponding reference blocks, determine whether to apply the LIC process to the multiple parts based on whether a Local Illumination Compensation (LIC) process is applied to the encoded blocks.

[0009] In a fifth aspect, according to some exemplary embodiments, an electronic device is provided, the electronic device comprising: a memory storing an instruction set; and one or more processors configured to execute the instruction set, such that the one or more processors perform a method of decoding a bitstream according to a first aspect to output one or more images of a video stream.

[0010] In a sixth aspect, according to some exemplary embodiments, an electronic device is provided, the electronic device comprising: a memory storing an instruction set; and one or more processors configured to execute the instruction set, such that the one or more processors perform a method for encoding a video sequence into a bitstream according to the second aspect.

[0011] In a seventh aspect, according to some exemplary embodiments, a non-transitory computer-readable storage medium is provided for storing a video bitstream, which, when decoded by a processor, causes the processor to perform a method for decoding the bitstream according to a first aspect to output one or more images of the video stream.

[0012] In an eighth aspect, according to some exemplary embodiments, a non-transitory computer-readable storage medium is provided for storing a video sequence of video, which, when encoded by a processor, causes the processor to perform a method for encoding the video sequence into a bitstream according to the second aspect.

[0013] In a ninth aspect, according to some exemplary embodiments, a computer program product is provided, the computer program product comprising: computer program instructions that enable a computer to perform a method of decoding a bitstream according to the first aspect to output one or more images of a video stream.

[0014] In a tenth aspect, according to some exemplary embodiments, a computer program product is provided, the computer program product comprising: computer program instructions that enable a computer to perform a method for encoding a video sequence into a bitstream according to the second aspect.

[0015] In an eleventh aspect, according to some exemplary embodiments, a computer program is provided that enables a computer to perform a method of decoding a bitstream according to the first aspect to output one or more images of a video stream.

[0016] In a twelfth aspect, according to some exemplary embodiments, a computer program is provided that enables a computer to perform the method of encoding a video sequence into a bitstream according to the second aspect. Attached Figure Description

[0017] Embodiments and aspects of this disclosure are illustrated in the following detailed description and accompanying drawings. Various features shown in the figures are not drawn to scale.

[0018] Figure 1 This is a schematic diagram illustrating an exemplary system for preprocessing and encoding image data according to some embodiments of the present disclosure.

[0019] Figure 2A This is a schematic diagram illustrating an exemplary encoding process of a hybrid video encoding system consistent with embodiments of this disclosure.

[0020] Figure 2B This is a schematic diagram illustrating another exemplary encoding process of a hybrid video encoding system consistent with embodiments of this disclosure.

[0021] Figure 3A This is a schematic diagram illustrating an exemplary decoding process of a hybrid video coding system consistent with embodiments of this disclosure.

[0022] Figure 3B This is a schematic diagram illustrating another exemplary decoding process of a hybrid video coding system consistent with embodiments of the present disclosure.

[0023] Figure 4 This is a block diagram of an exemplary apparatus for preprocessing or encoding image data according to some embodiments of the present disclosure.

[0024] Figure 5 This is a schematic diagram illustrating an exemplary local illumination compensation (LIC) template according to some embodiments of the present disclosure.

[0025] Figure 6 This is a schematic diagram illustrating geometric partition mode (GPM) segmentation according to some embodiments of the present disclosure, where the segments are grouped at the same angle.

[0026] Figure 7 This is a schematic diagram illustrating the selection of a unidirectional predicted motion vector (MV) for a geometric partitioning pattern according to some embodiments of the present disclosure.

[0027] Figure 8 This illustrates the generation of bending weights using a geometric partitioning pattern according to some embodiments of the present disclosure. An exemplary schematic diagram.

[0028] Figure 9 This is a schematic diagram illustrating five exemplary fusion region sizes of the GPM in an Enhanced Compression Model (ECM) according to some embodiments of the present disclosure.

[0029] Figure 10 This is a schematic diagram illustrating exemplary edges in a GPM partitioning pattern template according to some embodiments of the present disclosure.

[0030] Figure 11A -D is a schematic diagram illustrating GPM employing inter-frame prediction and intra-frame prediction according to some embodiments of the present disclosure.

[0031] Figure 12 This is an exemplary schematic diagram illustrating angular weighted prediction (AWP) weighted prediction according to some embodiments of the present disclosure.

[0032] Figure 13 This is a schematic diagram illustrating eight exemplary intra-frame prediction angles supported by AWP mode according to some embodiments of the present disclosure.

[0033] Figure 14 This is a schematic diagram illustrating seven different weight array settings in AWP mode according to some embodiments of the present disclosure.

[0034] Figure 15 This is a schematic diagram illustrating Overlapped Block Motion Compensation (OBMC) of coding unit (CU) boundaries according to some embodiments of the present disclosure.

[0035] Figure 16 This is a schematic diagram illustrating the spatial portion of an exemplary intra-block copy (IBC) filter according to some embodiments of the present disclosure.

[0036] Figure 17A This is a schematic diagram illustrating an exemplary method for decoding a bitstream according to some embodiments of the present disclosure.

[0037] Figure 17B This illustrates some embodiments according to the present disclosure. Figure 17A A schematic diagram of the sub-steps of the exemplary method shown.

[0038] Figure 18A This is a schematic diagram illustrating an exemplary template for GPM-LIC according to some embodiments of the present disclosure.

[0039] Figure 18B This is a schematic diagram illustrating a template of an exemplary GPM-LIC according to some embodiments of the present disclosure.

[0040] Figure 19 This is a schematic diagram illustrating an exemplary multi-model LIC according to some embodiments of the present disclosure.

[0041] Figure 20A This is a schematic diagram illustrating an exemplary method for encoding a video sequence into a bitstream according to some embodiments of the present disclosure.

[0042] Figure 20B This illustrates some embodiments according to the present disclosure. Figure 20A A schematic diagram of the sub-steps of the exemplary method shown. Detailed Implementation

[0043] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which, unless otherwise stated, the same numerals in different figures represent the same or similar elements. The embodiments set forth in the following description of the exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, these embodiments are merely examples of apparatuses and methods consistent with relevant aspects of this disclosure as set forth in the appended claims. Specific aspects of this disclosure are described in more detail below. In the event of any conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall prevail.

[0044] The Joint Video Experts Team (JVET) of the ITU-T Video Coding Expert Group (ITU-T VCEG) and the ISO / IEC Moving Picture Expert Group (ISO / IEC MPEG) is currently developing the Versatile Video Coding (VVC / H.266) standard. The VVC standard aims to double the compression efficiency of its predecessor, the High Efficiency Video Coding (HEVC / H.265) standard. In other words, VVC aims to achieve the same subjective quality using half the bandwidth of HEVC / H.265.

[0045] To achieve this goal, since 2015, JVET has been continuously developing technologies that surpass HEVC using the Joint Exploration Model (JEM) reference software. With the incorporation of various coding techniques into JEM, it has achieved significantly higher coding performance than HEVC. In October 2017, VCEG and MPEG issued a joint call for proposals (CfP), officially launching the development of a next-generation video compression standard that surpasses HEVC. Responses to the CfP were evaluated at the JVET meeting in San Diego in April 2018, and formal development of the VVC standard began in April 2018.

[0046] Since April 2018, the VVC standard has progressed smoothly and continues to incorporate more coding technologies to provide better compression performance. VVC adopts the hybrid video coding system used in modern video compression standards such as HEVC, H.264 / AVC, MPEG2, and H.263.

[0047] Figure 1 This is a block diagram illustrating a system 100 for preprocessing and encoding image data according to some of the disclosed embodiments. Image data may include images (also called “pictures” or “frames”), multiple images, or video. Images are still images. Multiple images may be spatially or temporally related or unrelated. Video is a set of images arranged in chronological order.

[0048] like Figure 1 As shown, system 100 includes source device 120, which provides encoded video data for subsequent decoding by target device 140. Consistent with the disclosed embodiments, each of source device 120 and target device 140 may include any of a variety of devices, including desktop computers, laptops (e.g., notebook computers), servers, tablets, set-top boxes, mobile phones, vehicles, cameras, image sensors, robots, televisions, wearable devices (e.g., smartwatches or wearable cameras), display devices, digital media players, video game consoles, video streaming devices, etc. Source device 120 and target device 140 may support wireless or wired communication.

[0049] refer to Figure 1The source device 120 may include an image / video preprocessor 122, an image / video encoder 124, and an output interface 126. The target device 140 may include an input interface 142, an image / video decoder 144, and one or more machine vision applications 146. The image / video preprocessor 122 preprocesses image data, i.e., one or more images or one or more videos, and generates an input bitstream for the image / video encoder 124. The image / video encoder 124 encodes the input bitstream and outputs an encoded bitstream 162 via the output interface 126. The encoded bitstream 162 is transmitted through a communication medium 160 and received by the input interface 142. The image / video decoder 144 then decodes the encoded bitstream 162 to generate decoded data, which can be used by the machine vision application 146.

[0050] More specifically, source device 120 may also include various devices (not shown) for providing source image data to be preprocessed by image / video preprocessor 122. The devices for providing the source image data may include image / video acquisition devices, such as cameras, image / video archives or storage devices containing previously acquired images / videos, or image / video feed interfaces for receiving images / videos from image / video content providers.

[0051] Image / video encoder 124 and image / video decoder 144 can each be implemented as any of a variety of suitable encoder or decoder circuit systems, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When encoding or decoding is partially implemented in software, image / video encoder 124 or image / video decoder 144 may store instructions for software in a suitable, non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform techniques consistent with this disclosure. Image / video encoder 124 or image / video decoder 144 may be included in one or more encoders or decoders, which may be integrated as part of a composite encoder / decoder (CODEC) in the respective device.

[0052] The image / video encoder 124 and image / video decoder 144 can operate according to any video coding standard, such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), or Versatile Video Coding (VVC). AOMedia Video 1 (AOMediaVideo 1, AV1), Joint Photographic Experts Group (JPEG), Dynamic Image Special Family Group (Moving Picture Experts Group, MPEG, etc.). Alternatively, the image / video encoder 124 and image / video decoder 144 can be custom devices that do not conform to existing standards. Although Figure 1 Not shown, but in some embodiments, both the image / video encoder 124 and the image / video decoder 144 may be integrated with the audio encoder and decoder, and may include suitable multiplexer-demultiplexer (MUX-DEMUX) units, or other hardware and software, to process the encoding of audio and video in a common data stream or separate data streams.

[0053] Output interface 126 may include any type of medium or device capable of transmitting the encoded bit stream 162 from source device 120 to target device 140. For example, output interface 126 may include a transmitter or transceiver configured to transmit the encoded bit stream 162 directly from source device 120 to target device 140 in real time. The encoded bit stream 162 may be modulated according to communication standards such as wireless communication protocols and transmitted to target device 140.

[0054] Communication medium 160 may include transient media, such as wireless broadcasting or wired network transmission. For example, communication medium 160 may include radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cables). Communication medium 160 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. In some embodiments, communication medium 160 may include a router, a switch, a base station, or any other device that may be used to facilitate communication from source device 120 to target device 140. For example, a network server (not shown) may receive encoded bitstream 162 from source device 120 and provide encoded bitstream 162 to target device 140, for example, via network transmission.

[0055] The communication medium 160 may also be in the form of a storage medium (e.g., a non-transitory storage medium), such as a hard disk, flash drive, optical disk, digital video optical disk, Blu-ray disc, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded image data. In some embodiments, a computing device of a media production facility (e.g., an optical disk imprinting facility) may receive encoded image data from source device 120 and produce an optical disk containing the encoded video data.

[0056] Input interface 142 may include any type of medium or device capable of receiving information from communication medium 160. The received information includes an encoded bitstream 162. For example, input interface 142 may include a receiver or transceiver configured to receive the encoded bitstream 162 in real time.

[0057] Machine vision application 146 includes various hardware and / or software for using decoded image data generated by image / video decoder 144. For example, machine vision application 146 may include a display device for displaying the decoded image data to a user, and may include any of a variety of display devices, such as cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light emitting diode (OLED) display, or other types of display devices. As another example, machine vision application 146 may include one or more processors configured to use the decoded image data to perform various machine vision applications, such as object recognition and tracking, face recognition, image matching, image / video search, augmented reality, robot vision and navigation, autonomous driving, 3D structure construction, stereo mapping, motion tracking, etc.

[0058] Next, combined Figure 2A-2B and Figures 3A-3B This describes exemplary image data encoding and decoding techniques.

[0059] Figure 2A A schematic diagram of an example encoding process 200A consistent with embodiments of this disclosure is shown. For example, the encoding process 200A may be performed by an encoder, such as... Figure 1 Image / video encoder 124 in the example. Figure 2AAs shown, the encoder can encode video sequence 202 into video bitstream 228 according to process 200A. Video sequence 202 may include a set of images (referred to as "raw images") arranged in chronological order. Each raw image of video sequence 202 may be divided by the encoder into multiple basic processing units, multiple basic processing subunits, or multiple regions for processing. In some embodiments, the encoder may perform process 200A at the level of basic processing units for each raw image of video sequence 202. For example, the encoder may perform process 200A iteratively, wherein the encoder may encode basic processing units in one iteration of process 200A. In some embodiments, the encoder may perform process 200A in parallel for multiple regions of each raw image of video sequence 202.

[0060] exist Figure 2A In this process, the encoder feeds the basic processing units (referred to as "basic processing units (BPUs)") of the raw image of the video sequence 202 to the prediction stage 204 to generate prediction data 206 and prediction BPU 208. The encoder can subtract prediction BPU 208 from the raw BPU to generate residual BPU 210. The encoder can feed residual BPU 210 to the transform stage 212 and the quantization stage 214 to generate quantization transform coefficients 216. The encoder can feed prediction data 206 and quantization transform coefficients 216 to the binary encoding stage 226 to generate video bitstream 228. Components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 can be referred to as the "forward path". During process 200A, after quantization phase 214, the encoder can feed quantization transform coefficients 216 to inverse quantization phase 218 and inverse transform phase 220 to generate reconstruction residual BPU 222. The encoder can add reconstruction residual BPU 222 to prediction BPU 208 to generate prediction reference 224, which is used in prediction phase 204 for the next iteration of process 200A. Components 218, 220, 222, and 224 of process 200A can be referred to as a "reconstruction path." The reconstruction path can be used to ensure that both the encoder and the decoder use the same reference data for prediction.

[0061] The encoder can iteratively execute process 200A to encode each raw BPU of the raw image (in the forward path) and generate prediction reference 224 for encoding the next raw BPU of the raw image (in the reconstruction path). After encoding all raw BPUs of the raw image, the encoder can continue to encode the next image in the video sequence 202.

[0062] Referring to process 200A, the encoder can receive a video sequence 202 generated by a video acquisition device (e.g., a camera). As used herein, the term "receive" can refer to any action of receiving, inputting, acquiring, retrieving, obtaining, reading, accessing, or otherwise inputting data.

[0063] In prediction phase 204, during the current iteration, the encoder may receive the original BPU and prediction reference 224, and perform prediction operations to generate prediction data 206 and prediction BPU 208. Prediction reference 224 can be generated from the reconstruction path of the previous iteration of process 200A. The purpose of prediction phase 204 is to reduce information redundancy by extracting prediction data 206 from prediction data 206 and prediction reference 224 that can be used to reconstruct the original BPU into prediction BPU 208.

[0064] Ideally, the predicted BPU 208 should be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 208 is typically slightly different from the original BPU. To record this difference, after generating the predicted BPU 208, the encoder can subtract it from the original BPU to generate the residual BPU 210. For example, the encoder can subtract the pixel value (e.g., grayscale or RGB value) of the predicted BPU 208 from the value of the corresponding pixel in the original BPU. Each pixel in the residual BPU 210 can have a residual value generated by this subtraction between the corresponding pixel values ​​of the original BPU and the predicted BPU 208. Compared to the original BPU, the predicted data 206 and the residual BPU 210 can have fewer bits, but they can be used to reconstruct the original BPU without significant quality degradation. Therefore, the original BPU is compressed.

[0065] To further compress the residual BPU 210, in the transform phase 212, the encoder can reduce the spatial redundancy of the residual BPU 210 by decomposing it into a two-dimensional set of "basic patterns," each basic pattern being associated with "transform coefficients." The basic patterns can have the same size (e.g., the size of the residual BPU 210). Each basic pattern can characterize a frequency-varying component of the residual BPU 210 (e.g., the frequency of brightness variation). No basic pattern can be reproduced by any combination of any other basic patterns (e.g., a linear combination). In other words, the decomposition decomposes the variation of the residual BPU 210 into the frequency domain. This decomposition is analogous to the discrete Fourier transform of a function, where the basic patterns are analogous to the basis functions of the discrete Fourier transform (e.g., trigonometric functions), and the transform coefficients are analogous to the coefficients associated with the basis functions.

[0066] Different transform algorithms can use different base modes. Various transform algorithms can be used in transform stage 212, such as discrete cosine transform, discrete sine transform, etc. The transform in transform stage 212 is reversible. That is, the encoder can recover the residual BPU 210 through the inverse operation of the transform (called the "inverse transform"). For example, to recover a pixel of the residual BPU 210, the inverse transform can be to multiply the value of the corresponding pixel in the base mode by the corresponding correlation coefficient and sum the products to produce a weighted sum. For video coding standards, both the encoder and decoder can use the same transform algorithm (and therefore the same base mode). Therefore, the encoder can only record the transform coefficients, and the decoder can reconstruct the residual BPU 210 based on the transform coefficients without receiving the base mode from the encoder. Compared to the residual BPU 210, the transform coefficients can have fewer bits, but they can be used to reconstruct the residual BPU 210 without significant quality degradation. Therefore, the residual BPU 210 is further compressed.

[0067] The encoder can further compress the transform coefficients in quantization phase 214. During the transform process, different fundamental modes can represent different change frequencies (e.g., brightness change frequencies). Because the human eye is generally better at identifying low-frequency changes, the encoder can ignore information about high-frequency changes without significantly degrading the decoding quality. For example, in quantization phase 214, the encoder can generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (called a "quantization parameter") and rounding the quotient to the nearest integer. After this operation, some transform coefficients of the high-frequency fundamental modes can be converted to zero, while the transform coefficients of the low-frequency fundamental modes can be converted to smaller integers. The encoder can ignore the zero-valued quantized transform coefficients 216, thereby further compressing the transform coefficients. The quantization process is also reversible, wherein the quantized transform coefficients 216 can be reconstructed into the transform coefficients in the inverse operation of quantization (called "inverse quantization").

[0068] Because the encoder ignores the remainder of such division in the rounding operation, the quantization stage 214 may be lossy. Typically, the quantization stage 214 may constitute the primary source of information loss in process 200A. The greater the information loss, the fewer bits the quantization transform coefficients 216 may require. To obtain different levels of information loss, the encoder can use different values ​​of the quantization parameters, or different values ​​of any other parameters of the quantization process.

[0069] In the binary encoding stage 226, the encoder may use binary encoding techniques to encode the prediction data 206 and the quantization transform coefficients 216, such as entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless or lossy compression algorithm. In some embodiments, in addition to the prediction data 206 and the quantization transform coefficients 216, the encoder may also encode other information in the binary encoding stage 226, such as the prediction mode used in the prediction stage 204, the parameters of the prediction operation, the transform type of the transform stage 212, the parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), etc. The encoder may use the output data of the binary encoding stage 226 to generate a video bitstream 228. In some embodiments, the video bitstream 228 may be further packaged for network transmission.

[0070] Referring to the reconstruction path of process 200A, in the inverse quantization stage 218, the encoder can perform inverse quantization on the quantized transform coefficients 216 to generate reconstructed transform coefficients. In the inverse transform stage 220, the encoder can generate a reconstruction residual BPU 222 based on the reconstructed transform coefficients. The encoder can add the reconstruction residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224 to be used in the next iteration of process 200A.

[0071] It should be noted that other variations of process 200A can be used to encode video sequence 202. In some embodiments, the stages of process 200A can be performed by the encoder in different orders. In some embodiments, one or more stages of process 200A can be combined into a single stage. In some embodiments, a single stage of process 200A can be divided into multiple stages. For example, transform stage 212 and quantization stage 214 can be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, process 200A may be omitted. Figure 2A One or more stages in the process.

[0072] Figure 2B A schematic diagram of another example encoding process 200B consistent with embodiments of this disclosure is shown. For example, the encoding process 200B may be performed by an encoder, such as... Figure 1 The image / video encoder 124 is used in the process. Process 200B can be modified from process 200A. For example, process 200B can be used by an encoder that conforms to hybrid video coding standards (e.g., H.26x series). Compared to process 200A, the forward path of process 200B additionally includes a mode decision stage 230, and divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B additionally includes a loop filtering stage 232 and a buffer 234.

[0073] Generally, prediction techniques can be categorized into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-image prediction or "intra-frame prediction") uses pixels from one or more encoded neighboring BPUs within the same image to predict the current BPU. That is, prediction reference 224 in spatial prediction can include the neighboring BPUs. Spatial prediction can reduce the inherent spatial redundancy of the image. Temporal prediction (e.g., inter-image prediction or "inter-frame prediction") uses regions from one or more encoded images to predict the current BPU. That is, prediction reference 224 in temporal prediction can include the encoded images. Temporal prediction can reduce the inherent temporal redundancy of the image.

[0074] Referring to process 200B, in the forward path, the encoder performs prediction operations in spatial prediction phase 2042 and temporal prediction phase 2044. For example, in spatial prediction phase 2042, the encoder may perform intra-frame prediction. For a given original BPU of an image being encoded, prediction reference 224 may include one or more adjacent BPUs that have already been encoded (in the forward path) and reconstructed (in the reconstruction path) in the same image. The encoder can generate a predicted BPU 208 by extrapolating adjacent BPUs. The extrapolation technique may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, etc. In some embodiments, the encoder may perform the extrapolation at the pixel level, for example, extrapolating the corresponding pixel value for each pixel of the predicted BPU 208. The adjacent BPU used for extrapolation can be located from various directions relative to the original BPU, such as in the vertical direction (e.g., at the top of the original BPU), the horizontal direction (e.g., to the left of the original BPU), the diagonal direction (e.g., at the lower left, lower right, upper left, or upper right of the original BPU), or any direction defined in the video coding standard used. For the intra-frame prediction, the prediction data 206 may include, for example, the location (e.g., coordinates) of the adjacent BPU used, the size of the adjacent BPU used, the extrapolation parameters, the orientation of the adjacent BPU used relative to the original BPU, etc.

[0075] In another example, during the temporal prediction phase 2044, the encoder may perform the inter-frame prediction. For a given original BPU of the current image, the prediction reference 224 may include one or more images (referred to as "reference images") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference images may be encoded and reconstructed on a BPU-by-BPU basis. For example, the encoder may add a reconstructed residual BPU 222 to the prediction BPU 208 to generate a reconstructed BPU. After all reconstructed BPUs of the same image have been generated, the encoder may generate a reconstructed image as the reference image. The encoder may perform a "motion estimation" operation to search for a matching region within a range (referred to as a "search window") of the reference image. The position of the search window in the reference image may be determined based on the position of the original BPU in the current image. For example, the search window may be centered at a position in the reference image that has the same coordinates as the original BPU in the current image and may extend outward by a predetermined distance. When the encoder identifies a region similar to the original BPU in the search window (e.g., using a pixel recursive algorithm, block matching algorithm, etc.), the encoder can determine such a region as a matching region. The matching region may have different specifications than the original BPU (e.g., less than, equal to, or greater than the original BPU, or have a different shape). Because the reference image and the current image are temporally separated on the timeline, the matching region can be considered to have "moved" to the position of the original BPU over time. The encoder can record the direction and distance of this movement as a "motion vector." When using multiple reference images, the encoder can search for matching regions for each reference image and determine the associated motion vector of the matching region. In some embodiments, the encoder can assign weights to the pixel values ​​of the matching region for each matching reference image.

[0076] The motion estimation can be used to identify various types of motion, such as translation, rotation, scaling, etc. For inter-frame prediction, the prediction data 206 may include, for example, the location (e.g., coordinates) of the matching region, the motion vector associated with the matching region, the number of reference images, the weights associated with the plurality of reference images, etc.

[0077] To generate the predicted BPU 208, the encoder may perform a "motion compensation" operation. This motion compensation can be used to reconstruct the predicted BPU 208 based on the predicted data 206 (e.g., motion vectors) and the predicted reference 224. For example, the encoder may move the matching region of the reference image according to the motion vectors, thereby predicting the original BPU of the current image. When using multiple reference images, the encoder may move the matching regions of the multiple reference images according to their respective motion vectors and average pixel values. In some embodiments, if the encoder has already assigned weights to the pixel values ​​of the matching regions of each matching reference image, the encoder may perform a weighted summation of the pixel values ​​of the moved matching regions.

[0078] In some embodiments, the inter-frame prediction can be unidirectional or bidirectional. Unidirectional inter-frame prediction can use one or more reference images in the same time direction relative to the current image. Unidirectional inter-frame prediction uses reference images preceding the current image. Bidirectional inter-frame prediction can use one or more reference images in two different time directions relative to the current image.

[0079] Still referring to the forward path of process 200B, after spatial prediction stage 2042 and temporal prediction stage 2044, in mode decision stage 230, the encoder can select a prediction mode (e.g., one of intra-frame prediction or inter-frame prediction) for the current iteration of process 200B. For example, the encoder can perform rate-distortion optimization techniques, wherein the encoder selects a prediction mode based on the bit rate of a candidate prediction mode and the distortion of the reference image reconstructed under the candidate prediction mode to minimize the value of the cost function. Based on the selected prediction mode, the encoder can generate a corresponding prediction BPU 208 and prediction data 206.

[0080] In the reconstruction path of process 200B, if an intra-frame prediction mode has been selected in the forward path, the encoder can directly feed the prediction reference 224 (e.g., the current BPU that has been encoded and reconstructed in the current image) to the spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU in the current image) after generating the prediction reference 224. If an inter-frame prediction mode has been selected in the forward path, the encoder can feed the prediction reference 224 (e.g., the current image where all BPUs have been encoded and reconstructed) to the loop filtering stage 232 after generating the prediction reference 224, whereby the encoder can apply loop filtering to the prediction reference 224 to reduce or eliminate distortions (e.g., blockiness) introduced by the inter-frame prediction. The encoder can apply various loop filtering techniques in the loop filtering stage 232, such as deblocking, sample adaptive shifting, adaptive loop filtering, etc. The loop-filtered reference image can be stored in buffer 234 (or “decoded image buffer”) for subsequent use (e.g., as an inter-frame prediction reference image for future images of video sequence 202). The encoder can store one or more reference images in buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the encoder can encode the loop-filter parameters (e.g., loop-filter strength), as well as the quantization transform coefficients 216, prediction data 206, and other information in the binary encoding stage 226.

[0081] In some embodiments, the input video sequence 202 is processed block by block according to the encoding process 200B. In VVC, the coded tree unit (CTU) is the largest block unit and can be as large as 128×128 luma samples (plus the corresponding chroma samples according to the chroma format). The CTU can be further divided into coded units (CUs) using quadtrees, binary trees, or ternary trees. At the leaf nodes of the segmentation structure, encoded information such as the encoding mode (intra-frame mode or inter-frame mode), motion information (reference index, motion vector difference, etc.) in the case of inter-frame coding, and quantization transform coefficients 216 are sent. If intra-frame prediction (also known as spatial prediction) is used, the current block is predicted using spatially adjacent samples. If inter-frame prediction (also known as temporal prediction or motion-compensated prediction) is used, the current block is predicted using samples from an encoded image called a reference image. Inter-frame prediction can use unidirectional or bidirectional prediction. In unidirectional prediction, only one motion vector pointing to a reference image is used to generate the predicted signal for the current block; while in bidirectional prediction, two motion vectors, each pointing to its own reference image, are used to generate the predicted signal for the current block. The motion vectors and reference indices are sent to the decoder to identify where one or more predicted signals for the current block originate. After intra-frame or inter-frame prediction, mode decision stage 230 selects the optimal prediction mode for the current block, for example, based on a rate-distortion optimization method. Based on the optimal prediction mode, a prediction BPU 208 is generated and subtracted from the input video block.

[0082] Still referencing Figure 2B The prediction residual BPU 210 is sent to transform stage 212 and quantization stage 214 to generate quantized transform coefficients 216. The quantized transform coefficients 216 are then dequantized in inverse quantization stage 218 and inverse transformed in inverse transform stage 220 to obtain reconstructed residual BPU 222. The prediction BPU 208 and reconstructed residual BPU 222 are added together to form a prediction reference 224 prior to loop filtering, which provides reference samples for intra-frame prediction. Loop filtering (e.g., deblocking, sample adaptive offset (SAO), and adaptive loop filter (ALF)) can be applied to the prediction reference 224 in loop filtering stage 232 to form the reconstructed block, which is stored in buffer 234 and used to provide reference samples for inter-frame prediction. The encoding information generated in the mode decision stage 230, such as encoding mode (intra-frame or inter-frame prediction), intra-frame prediction mode, motion information, quantized residual coefficients, etc., is sent to the binary encoding stage 226 to further reduce the bit rate before being packaged into the output video bitstream 228.

[0083] Figure 3A A schematic diagram of an example decoding process 300A consistent with embodiments of this disclosure is shown. For example, the decoding process 300A may be performed by a decoder, such as... Figure 1 The image / video decoder 144 in the process 300A can be corresponding to... Figure 2A The decompression process of compression process 200A. In some embodiments, process 300A may be similar to the reconstruction path described in process 200A. Decoder (e.g., Figure 1 The image / video decoder 144 in the image / video decoder can decode the video bitstream 228 into a video stream 304 according to process 300A. The video stream 304 can be very similar to the video sequence 202. However, due to information loss during compression and decompression (e.g., Figure 2A-2B In the quantization stage 214, video stream 304 typically differs from video sequence 202. Similar to... Figure 2A-2B In processes 200A and 200B, the decoder can perform process 300A at the level of a basic processing unit (BPU) for each image encoded in the video bitstream 228. For example, the decoder can perform process 300A iteratively, wherein the decoder can decode one basic processing unit in one iteration of process 300A. In some embodiments, the decoder can perform process 300A in parallel for multiple regions of each image encoded in the video bitstream 228.

[0084] exist Figure 3A In this process, the decoder may feed a portion of the video bitstream 228 associated with a basic processing unit (referred to as a "coded BPU") of the encoded image to the binary decoding stage 302. In the binary decoding stage 302, the decoder may decode the portion into prediction data 206 and quantization transform coefficients 216. The decoder may feed the quantization transform coefficients 216 to the inverse quantization stage 218 and the inverse transform stage 220 to generate a reconstructed residual BPU 222. The decoder may feed the prediction data 206 to the prediction stage 204 to generate a prediction BPU 208. The decoder may add the reconstructed residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224. In some embodiments, the prediction reference 224 may be stored in a buffer (e.g., a decoded image buffer in computer memory). The decoder may feed the prediction reference 224 to the prediction stage 204 for performing a prediction operation in the next iteration of process 300A.

[0085] The decoder can iteratively execute process 300A to decode each encoded BPU of the encoded image and generate a prediction reference 224 for encoding the next encoded BPU of the encoded image. After decoding all encoded BPUs of the encoded image, the decoder can output the image to video stream 304 for display and continue decoding the next encoded image in video bitstream 228.

[0086] In binary decoding stage 302, the decoder may perform the inverse operation of the binary encoding technique used by the encoder (e.g., entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in addition to the prediction data 206 and quantization transform coefficients 216, the decoder may also decode other information in binary decoding stage 302, such as prediction mode, parameters of the prediction operation, transform type, parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), etc. In some embodiments, if the video bitstream 228 is transmitted over the network in the form of data packets, the decoder may unpack the video bitstream 228 before feeding it to binary decoding stage 302.

[0087] Figure 3B A schematic diagram of another example decoding process 300B consistent with embodiments of this disclosure is shown. For example, the decoding process 300B may be performed by a decoder, such as... Figure 1 The image / video decoder 144 is included. Process 300B can be modified from process 300A. For example, process 300B can be used by a decoder conforming to a hybrid video coding standard (e.g., H.26x series). Compared to process 300A, process 300B further divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and additionally includes a loop filtering stage 232 and a buffer 234.

[0088] In process 300B, for the encoded basic processing unit (referred to as the "current BPU") of the encoded image being decoded (referred to as the "current image"), the prediction data 206 decoded by the decoder from the binary decoding stage 302 can include various types of data depending on the prediction mode used by the encoder to encode the current BPU. For example, if the encoder uses intra-frame prediction to encode the current BPU, the prediction data 206 can include a prediction mode indicator (e.g., a flag value) indicating the intra-frame prediction, parameters of the intra-frame prediction operation, etc. The parameters of the intra-frame prediction operation can include, for example, the positions (e.g., coordinates) of one or more neighboring BPUs used as references, the size of the neighboring BPUs, extrapolation parameters, the orientation of the neighboring BPUs relative to the original BPU, etc. For another example, if the encoder uses inter-frame prediction to encode the current BPU, the prediction data 206 can include a prediction mode indicator (e.g., a flag value) indicating the inter-frame prediction, parameters of the inter-frame prediction operation, etc. The parameters of the inter-frame prediction operation may include, for example, the number of reference images associated with the current BPU, the weights associated with the reference images respectively, the positions (e.g., coordinates) of one or more matching regions in each reference image, and one or more motion vectors associated with the matching regions respectively.

[0089] Based on the prediction mode indicator, the decoder can determine whether to perform spatial prediction (e.g., intra-frame prediction) in the spatial prediction phase 2042 or temporal prediction (e.g., inter-frame prediction) in the temporal prediction phase 2044. Figure 2B The details of performing such spatial or temporal predictions are described herein and will not be repeated below. After performing such spatial or temporal predictions, the decoder can generate a prediction BPU 208. The decoder can then add the prediction BPU 208 and the reconstruction residual BPU 222 to generate a prediction reference 224, as shown below. Figure 3A As described.

[0090] In process 300B, the decoder can feed the prediction reference 224 to either spatial prediction stage 2042 or temporal prediction stage 2044 for performing prediction operations in the next iteration of process 300B. For example, if the current BPU is decoded using the intra-frame prediction in spatial prediction stage 2042, the decoder can feed the prediction reference 224 directly to spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU of the current image) after generating the prediction reference 224 (e.g., the decoded current BPU). If the current BPU is decoded using the inter-frame prediction in temporal prediction stage 2044, the encoder can feed the prediction reference 224 to loop filtering stage 232 to reduce or eliminate distortion (e.g., blockiness) after generating the prediction reference 224 (e.g., a reference image where all BPUs have been decoded). The decoder can, as shown in... Figure 2B The method described herein applies loop filtering to prediction reference 224. The loop-filtered reference image can be stored in buffer 234 (e.g., a decoded image buffer in computer memory) for subsequent use (e.g., as an inter-frame prediction reference image for a future encoded image of video bitstream 228). The decoder can store one or more reference images in buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the prediction data 206 may also include parameters of the loop filtering (e.g., loop filtering strength) when the prediction mode indicator of the prediction data 206 indicates that inter-frame prediction has been used to encode the current BPU.

[0091] Return to reference Figure 1 The image / video preprocessor 122, the image / video encoder 124, and the image / video decoder 144 can all be implemented as any suitable hardware, software, or combination thereof. Figure 4 This is a block diagram of an example apparatus 400 for processing image data, consistent with embodiments of this disclosure. For example, apparatus 400 may be a preprocessor, encoder, or decoder. Figure 4As shown, device 400 may include processor 402. When processor 402 executes the instructions described herein, device 400 may become a dedicated machine for preprocessing, encoding, and / or decoding image data. Processor 402 may be any type of circuit system capable of manipulating or processing information. For example, processor 402 may include any number of central processing units (or “CPU”), graphics processing units (or “GPU”), neural processing units (“NPU”), microcontroller units (“MCU”), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLAs), programmable array logic (PALs), generic array logic (GALs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), system-on-chip (SoCs), application-specific integrated circuits (ASICs), and any combination thereof. In some embodiments, processor 402 may also be a set of processors grouped into individual logical components. For example, such as Figure 4 As shown, processor 402 may include multiple processors, including processor 402a, processor 402b and processor 402n.

[0092] The device 400 may also include a memory 404 configured to store data (e.g., instruction sets, computer code, intermediate data, etc.). For example, such as Figure 4As shown, the stored data may include program instructions (e.g., program instructions for implementing the stages in processes 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequence 202, video bitstream 228, or video stream 304). Processor 402 can access the program instructions and data for processing (e.g., via bus 410) and execute the program instructions to perform operations or manipulations on the data for processing. Memory 404 may include a high-speed random access memory device or a non-volatile memory device. In some embodiments, memory 404 may include any combination of any number of random-access memory (RAM), read-only memory (ROM), optical discs, magnetic disks, hard disks, solid-state drives, flash drives, security digital (SD) cards, memory sticks, compact flash (CF) cards, etc. Memory 404 may also be a group of memories grouped into a single logical component. Figure 4 (Not shown in the text).

[0093] Bus 410 may be a communication device for transmitting data between components within device 400, such as an internal bus (e.g., CPU-memory bus), an external bus (e.g., a Universal Serial Bus port, a Peripheral Component Interconnect Fast Port), etc.

[0094] For ease of explanation and to avoid ambiguity, processor 402 and other data processing circuitry are collectively referred to as "data processing circuitry" in this disclosure. The data processing circuitry may be implemented entirely as hardware, or as a combination of software, hardware, or firmware. Furthermore, the data processing circuitry may be a single, independent module, or may be wholly or partially integrated into any other component of device 400.

[0095] Device 400 may also include a network interface 406 to provide wired or wireless communication with a network (e.g., the Internet, intranet, local area network, mobile communication network, etc.). In some embodiments, network interface 406 may include any combination of any number of network interface controllers (NICs), radio frequency (RF) modules, transceivers, transceivers, modems, routers, gateways, wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (“NFC”) adapters, cellular network chips, etc.

[0096] In some embodiments, the device 400 may further include a peripheral interface 408 to provide connectivity to one or more peripheral devices. For example... Figure 4As shown, the peripheral devices may include, but are not limited to, cursor control devices (e.g., mouse, touchpad, or touchscreen), keyboards, displays (e.g., cathode ray tube displays, liquid crystal displays, or light-emitting diode displays), video input devices (e.g., cameras or input interfaces coupled to video archives), etc.

[0097] It should be noted that the video codec (e.g., the codec for executing processes 200A, 200B, 300A, or 300B) can be implemented as any combination of any software or hardware modules in device 400. For example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more software modules of device 400, such as program instructions that can be loaded into memory 404. As another example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more hardware modules of device 400, such as dedicated data processing circuitry (e.g., FPGA, ASIC, NPU, etc.).

[0098] In July 2020, the first version of the VVC standard was finalized and officially released as an international standard. Subsequently, JVET began exploring new coding tools to further improve the coding performance of the VVC standard (e.g., the April 2020 version). In January 2021, the Enhanced Compression Model (ECM) (e.g., ECM-21.0) was proposed and used as the foundation for developing new software that goes beyond the VVC standard. In the following text, unless otherwise stated, VVC / ECM may refer to the corresponding version described herein. It is understood that the methods described herein are also applicable to subsequent versions of VVC / ECM (if any).

[0099] Local Illumination Compensation (LIC) is an inter-frame prediction technique used to model the local illumination variation between the current block and its reference block as a function of the local illumination variation between the current block template and the reference block template. The parameters of this function can be a scaling factor. α and an offset β This can be represented as follows: these two parameters constitute a linear equation for compensating for changes in illumination. The parameters α and β can be based on, for example... Figure 5 The current template and reference template shown are used for derivation. The samples in the current template of the current image can be reconstructed samples available in the adjacent row above and the adjacent column to the left of the current block (which may also be referred to as a coding unit (CU) in this disclosure). The samples in the reference template of the reference image can be generated by motion compensation using the motion vector of the current block. In some embodiments, a subset of the samples in the current template and the reference template can be used to derive LIC parameters. These derived parameters are then applied to adjust the predicted values ​​of the samples in the current block, as follows: (1) in, Indicates the coordinate position of the sample within the current block; Indicates the predicted value of the sample after motion compensation; Indicates the adjusted predicted value of the sample after applying the LIC linear model; and α and β It is a LIC parameter.

[0100] Due to LIC parameters α and β Both the encoder and decoder can derive these parameters based on the current and reference templates, therefore these two parameters do not require signaling overhead and only need to be performed at the CU level for advanced motion vector prediction. AMVP The mode signaling transmits a LIC flag to identify the use of the LIC. For merge mode coding (CU), the LIC flag can be inherited from the coded blocks based on the merge candidate index.

[0101] In some disclosed embodiments, the LIC can be extended to bidirectional prediction blocks. Specifically, two different linear models are derived to compensate for illumination variations between the current block and its two reference blocks. The final bidirectional prediction value of the current block is then calculated by combining the two LIC-adjusted unidirectional prediction blocks, i.e.: (2) as well as (3) (4) in, and as well as and These indicate the scaling factor and offset in L0 and L1, respectively; The weights applied when merging the two unidirectional prediction blocks are indicated (by the BCW index at the CU level). Similar to the current LIC design, for AMVP bidirectional prediction CUs, a control flag is signaled to indicate whether the LIC is enabled / disabled; for merge-mode encoded CUs (including AMVP-merge mode), this flag is inherited from an already encoded block. Furthermore, the LIC is disabled when decoder-side motion vector refinement (DMVR) (including multi-round DMVR, adaptive DMVR, and affine DMVR) and bidirectional optical flow (BDOF) are applied.

[0102] To reuse the existing linear model derivation of LIC, an iterative method is employed to alternately derive the L0 and L1 linear models. Specifically, given the two MVs of the current block, assume... and It is the template of the current block. The method first derives the two prediction results. and The L0 linear model yields... and The minimum difference between them; then the L1 linear model can be calculated ( and The L1 linear model will minimize The difference between the updated template and the actual template. Finally, the L0 linear model is corrected again in the same way.

[0103] For coded blocks employing combined intra- and inter-prediction (CIIP), the LIC flag can be inherited from the coded blocks based on the merge candidate index when calculating inter-prediction. In some embodiments, when the difference between the POC index of the L0 reference image and the POC index of the current image is 1, the LIC flag can only be used for inheritance in low-latency images.

[0104] Traditional LIC methods are not suitable for Geometric Partition Pattern (GPM) coded blocks.

[0105] In VVC, inter-frame prediction supports Geometric Partitioning (GPM). This GPM is a merging mode transmitted via CU-level flag signaling. A total of 64 GPM partitioning modes are supported for each possible CU size.

[0106] When using the GPM mode, according to the GPM partitioning pattern, a CU is divided into two parts by a geometrically positioned straight line. Figure 6 The position of the dividing line is mathematically derived from the specific division angle and offset parameters. Table 1 shows the relationship between the GPM partitioning mode (merge_gpm_partition_idx) and the GPM partitioning angle (angleIdx). Each part of the geometric partition in the CU uses its own motion information for inter-frame prediction; in VVC, each partition only allows unidirectional prediction, that is, each part has a motion vector and a reference index. Unidirectional prediction motion constraints are applied to ensure that, as with traditional bidirectional prediction, each CU only needs two motion-compensated predictions.

[0107] If the current CU is using GPM mode, a geometric partition index indicating the partitioning mode (corresponding to an angle and an offset) and two merge indices (one for each partition) are also signaled. The value of the largest GPM candidate size is explicitly signaled in SPS, specifying the syntax binarization rules for the GPM merge indices. After prediction is completed for each part of the geometric partition, the predicted sample values ​​along the geometric partition boundaries are adjusted by a fusion process using adaptive weights. This is the prediction signal for the entire CU, and as with other prediction modes, the transformation and quantization processes are applied to the entire CU. Finally, the motion field of the CU predicted using GPM mode is stored.

[0108] Table 1: Relationship between GPM partitioning mode (merge_gpm_partition_idx) and GPM partitioning angle (angleIdx).

[0109] Next, we will describe the construction of the GPM candidate list.

[0110] In some embodiments, a one-way prediction candidate list construction method is employed. The one-way prediction candidate list used in VVC is directly derived from the merge candidate list constructed for the regular merging mode. Let n denote the index of the one-way predicted motion in the geometric one-way prediction candidate list. The LX motion vector of the nth extended merge candidate (where X indicates the parity of n) is used as the nth one-way predicted motion vector in the geometric partitioning mode. These motion vectors in... Figure 7 The value is marked with "x". If there is no LX motion vector corresponding to the nth extended merge candidate, the L(1 - X) motion vector of the same candidate can be used instead as the unidirectional predicted motion vector in the geometric partitioning mode.

[0111] In ECM, a novel one-way prediction candidate list is proposed, and its construction order is as follows: - The interleaved L0 MV candidates and L1 MV candidates are directly derived from the regular merged candidate list, with L0 MV candidates having higher priority than L1 MV candidates. Based on the current CU size, a pruning method with an adaptive threshold is used to remove redundant MV candidates.

[0112] - The interleaved L1 MV candidates and L0 MV candidates are further derived directly from the regular merged candidate list, where L1 MV candidates have a higher priority than L0 MV candidates. The same pruning method with the aforementioned adaptive threshold is also used to remove redundant MV candidates.

[0113] - Fill the zero-value MV candidate list until the GPM candidate list is full.

[0114] In some embodiments, bidirectional prediction GPM is used. In ECM, the current GPM design is extended to support the use of multiple bidirectional prediction MVs, meaning that each part of a GPM-coded block can be predicted using either unidirectional or bidirectional prediction. For smaller blocks, such as 8×8, 16×8, and 8×16 blocks, only unidirectional prediction is supported, and the aforementioned unidirectional prediction candidate list is used. For other larger data blocks, a new merge list (which may contain bidirectional prediction motions) is used. The new merge list is generated in the same way as the regular merge list, except that the threshold for controlling whether a candidate can be added to the list is increased.

[0115] In some embodiments, fusion is performed along geometrically defined boundaries.

[0116] Next, the fusion method in VVC is described. In VVC, after predicting the motion information of each part of the geometric partition, the two predicted signals are fused to derive samples located near the geometric partition boundary. The fusion weight of each position of the CU is derived based on the distance of each position to the geometric partition boundary.

[0117] Location The distance to the geometric boundary is derived as follows: (5) (6) (7) (8) in, These are indices of the geometric division angles and offsets, which depend on the geometric division index of the signaling transmission. Symbols and Depends on the angle index .

[0118] The weights of each part of the geometric partition are derived as follows: (9) (10) (11) The partIdx depends on the angle index. . Figure 8 The weights are shown in the figure. An example.

[0119] Next, the fusion method in ECM is described. In ECM, the fusion process is improved by adding four additional fusion region sizes (1 / 4, 1 / 2, 2 times, and 4 times the fusion region size in VVC, respectively), as follows: Figure 9 As shown. A CU-level flag used to identify the selected fusion region size is transmitted via signaling. Furthermore, extended weight precision is employed, where the maximum value of the weight is increased from 8 (in the VVC) to 32 to accommodate the expanded fusion region size.

[0120] Next, we describe GPM with motion vector refinement (GPM-MMVD). In ECM, GPM is extended by applying motion vector refinement to the unidirectional prediction MV. First, a flag is encoded for the CU signaling transmission to indicate whether the GPM MMVD mode is used. If the GPM MMVD mode is used, each part of the geometric partition in the CU can further determine whether to transmit MVD signaling. If MVD is transmitted for a certain geometric part, the motion of that part is further refined using the MVD information transmitted in the signaling transmission after a GPM merging candidate is selected.

[0121] The MVD transmits signaling in pairs of distance and direction. In GPM with MMVD (GPM-MMVD), nine candidate distances (¼ pixel, ½ pixel, 1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels) and eight candidate directions (four horizontal / vertical directions and four diagonal directions) are involved.

[0122] Next, we describe GPM with template matching (GPM-TM). In ECM, the template matching method is applied to GPM. When GPM mode is enabled for a CU, a signaling CU-level flag indicates whether template matching is applied to both geometric parts. Motion information for each geometric part is corrected using TM. When TM is selected, a template is constructed using neighboring samples from the left and top, or left and top, based on the partition angle, as shown in Table 2. The motion is then corrected by minimizing the difference between the template of the current block and the template in the reference image using the same search mode of the merging mode and disabling the half-pixel interpolation filter.

[0123] Table 2: Templates for the first and second geometric parts, where A represents the use of the upper adjacent sample, L represents the use of the left adjacent sample, and L+A represents the use of both the left and upper adjacent samples.

[0124] GPM-MMVD and GPM-TM are enabled only for one GPM CU. This is done by first signaling the GPM-MMVD syntax. When both GPM-MMVD control flags are equal to "false" (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is signaled to indicate whether the template matching is applied to both GPM partitions. Otherwise (if at least one GPM-MMVD flag is equal to "true"), the value of the GPM-TM flag is inferred to be "false".

[0125] Next, the GPM partition pattern reordering is described. In ECM, 64 GPM partition patterns can be reordered using template matching. Based on the motion information of the current GPM coded block, the template matching (TM) cost between the current template and the reference templates of each GPM partition pattern is calculated. Then, all GPM partition patterns are reordered in ascending order based on the TM cost. Instead of sending GPM partition patterns, an index is sent via signaling, using Golomb-Rice code, indicating the specific position of the GPM partition pattern in the reordering list.

[0126] The GPM partitioning reordering method involves two steps, executed after the reference templates for each of the two GPM partitions are generated in the CU, as follows: ● Extend the GPM partition boundary to the reference templates of the two GPM partition parts to obtain 64 reference templates, and calculate the corresponding TM cost for each of the 64 reference templates. ● The TM generation value based on the GPM partitioning pattern is reordered in ascending order, and then the top 32 partitioning patterns are marked as available partitioning patterns.

[0127] like Figure 10 As shown, the boundary of the template extends from the boundary of the current CU, but the GPM fusion process is not used in the template region that crosses the boundary.

[0128] After reordering in ascending order using TM cost, a signaling index is transmitted to indicate the GPM partitioning mode.

[0129] Next, the intra- and inter-frame modes of GPM are described. In GPM that combines inter- and intra-frame prediction, the final prediction samples are generated by weighting the inter-frame and intra-frame prediction samples for each GPM partition. The inter-frame prediction samples are obtained through inter-frame prediction, while the intra-frame prediction samples are obtained through an intra-prediction mode (IPM) candidate list and an index of the encoder signaling transmission. The size of the IPM candidate list is predefined as 3. Figures 11A to 11C As shown, the available IPM candidates are: a parallel angle mode (parallel mode) parallel to the GPM block boundary, a vertical angle mode (vertical mode) perpendicular to the GPM block boundary, and a planar mode. Furthermore, as... Figure 11D As shown, the use of intra-frame and intra-frame prediction GPM is limited to reduce the signaling overhead of IPM and avoid increasing the size of the intra-frame prediction circuitry in the hardware decoder. Furthermore, direct storage of motion vectors and IPM is introduced in the GPM fusion region to further improve coding performance.

[0130] In some implementations, an IPM export method based on DIMD and adjacent patterns is employed, where parallel patterns are registered first. Therefore, if no identical IPM candidates are found in the list, up to two IPM candidates exported from the decoder's intra-frame pattern export (DIMD) method and / or adjacent blocks can be registered. Regarding adjacent pattern export, a maximum of five adjacent blocks can be placed.

[0131] GPM-intraframe can be combined with GPM with merge with motion vector difference (GPM-MMVD). TIMD is used as an IPM candidate for GPM-intraframe to further improve coding performance. Parallel modes can be registered first, followed by TIMD, DIMD, and IPM candidates for adjacent blocks.

[0132] Next, the motion field storage for the geometric partitioning pattern is described. Mv1 from the first part of the geometric partition, Mv2 from the second part of the geometric partition, and the combination Mv of Mv1 and Mv2 are all stored in the motion field of the geometric partitioning pattern encoding CU.

[0133] The type of motion vector stored at each independent location in the sports field is determined as follows: (12) Where motionIdx equals The result is obtained by recalculating from formula (1). `partIdx` depends on the angle index. .

[0134] If sType equals 0 or 1, then Mv1 or Mv2 is stored in the corresponding motion field; otherwise, if sType equals 2, a combination Mv of Mv1 and Mv2 is stored. The combined Mv is generated through the following process: If Mv1 and Mv2 come from different lists of reference images (one from L0 and the other from L1), then simply combining Mv1 and Mv2 will produce a bidirectional predicted motion vector.

[0135] Otherwise, if Mv1 and Mv2 come from the same list, only the unidirectional predicted motion Mv2 is stored.

[0136] Next, we describe Angle-Weighted Prediction (AWP). Similar to GPM in VVC, Audio Video Coding Standard 3 (AVS3) employs a tool called Angle-Weighted Prediction (AWP). The AVS3 video standard was developed by the AVS Working Group, which was established in China in 2002. AVS3's predecessors, AVS1 and AVS2, were released as Chinese national standards in 2006 and 2016, respectively. In AVS3, both skip and direct modes support angle-weighted prediction. The AWP mode is transmitted as either a skip or direct mode with a CU-level flag signaling. In the AWP mode, a motion vector candidate list is first constructed by deriving motion vectors from spatially adjacent blocks and a temporal motion vector predictor. This list contains five different unidirectional predicted motion vectors. Then, two unidirectional predicted motion vectors are selected from this candidate list to predict the current block. Unlike bidirectional prediction inter-frame modes, which apply the same weight to all samples, each sample encoded in AWP mode can have a different weight. The weights for each sample are predicted based on a weight array, where the values ​​range from 0 to 8. This weight prediction is similar to the process in intra-frame prediction, such as... Figure 12 As shown in the diagram. For each possible CU size ,in, The AWP mode supports a total of 56 different weights, including 8 intra-frame prediction angles (such as...). Figure 13 (as shown) and 7 different weight array settings (such as) Figure 14 (As shown). It should be noted that the AWP mode is directly signaled to the decoder without prediction. The AWP mode index is binarized using truncated binary; that is, indices 0 to 7 are encoded using 5 bits, while indices 8 to 55 are encoded using 6 bits.

[0137] Assume the two selected unidirectional predicted motion vectors are Mv0 and Mv1. By performing motion compensation using Mv0 and Mv1 respectively, two predicted blocks, P0 and P1, are obtained. The final predicted block P is calculated as follows: , (13) Here, w0 is the weight matrix derived using the weight prediction method described above.

[0138] After prediction is completed, the unidirectional predicted motion vector is stored in a 4×4 granularity. For each 4×4 cell, one of the two unidirectional predicted motion vectors is stored.

[0139] In some publicly disclosed embodiments, Overlapping Block Motion Compensation (OBMC) is an inter-frame coding tool for ECM. When OBMC is applied to a CU, it is performed against the top and left boundaries of the CU. Furthermore, when a CU is encoded using sub-CU modes (such as affine and DMVR modes), OBMC is further performed against the boundaries of each sub-CU, but the boundaries of the CU are not considered. To handle boundaries uniformly, OBMC is performed at the 4×4 sub-block level for all enabled boundaries. OBMC is applied to both luma and chroma components.

[0140] For the top and left CU boundaries, such as Figure 15 As shown, OBMC can be performed at the 4×4 sub-block level. When OBMC is applied to the current sub-block, in addition to using the current motion vector... In addition, if the current sub-block is located at the top boundary, the motion vector from the top adjacent sub-block is used. If the current sub-block is located on the left boundary, then the motion vector from the left adjacent sub-block is used. ,if Available and related to the current motion vector The motion vector is different. It is also used to derive the prediction block of the current sub-block. Based on the current motion vector The representation of is The predicted signal, and based on the adjacent motion vectors The representation of is The prediction signals are fused together to generate the final prediction signal for the current sub-block. .if equal If so, OBMC will not be performed on the current sub-block.

[0141] For sub-CU boundaries, OBMC can be performed at the 4×4 sub-block level, but CU boundaries are expected. For each sub-block, in addition to using the current motion vector, the motion vectors of four interconnected adjacent sub-blocks, if available and different from the current motion vector, are also used to derive the prediction block for the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal for the current sub-block.

[0142] In AMVP mode, a trusted signaling transmission carries a flag to indicate whether OBMC is performed for the CU. In skip and merge modes, no signaling is required, and OBMC is always performed.

[0143] In some publicly available embodiments, the following two adjustments can be used to better handle the interaction between the LIC and the OBMC: 1) It is proposed to apply the OBMC to inter-frame blocks that have applied the LIC. Furthermore, to achieve a better complexity / performance tradeoff, the OBMC is only applied to correct the predicted samples on the top and left boundaries of a LIC CU, while the OBMC on the boundaries of its internal sub-blocks is always disabled.

[0144] 2) In addition to using multiple MVs, it is proposed that when generating a prediction sample corresponding to a neighboring block for the current CU's OBMC, the LIC parameter of the neighboring block (when it is encoded by LIC) should also be taken into account. Therefore, the LIC parameter is stored at the CU level.

[0145] Intra-Block Copy (IBC) is a tool used in VVC. As is understood, IBC can significantly improve the encoding efficiency of screen content materials. Because IBC mode is implemented in a block-level encoding mode, block matching (BM) can be performed at the encoding end to find the optimal block vector (or motion vector) for each CU. Here, the block vector can be used to indicate the displacement from the current block to a reference block that has already been reconstructed within the current image. The luma block vector of the IBC-encoded CU has integer precision. The chroma block vector is also rounded to integer precision. When combined with AMVR, the IBC mode can switch between 1-pixel motion vector precision and 4-pixel motion vector precision. In addition to intra-frame or inter-frame prediction modes, IBC-encoded CUs are considered a third prediction mode. The IBC mode is suitable for CUs with a width and height of 64 luma samples or less.

[0146] At the encoder, hash-based motion estimation can be performed for IBC. The encoder performs RD checks on blocks with a width or height of no more than 16 luminance samples. For non-merging modes, a block vector search can be performed first using a hash-based search. If the hash search does not return valid candidates, a local search based on block matching will be performed.

[0147] In hash-based search, hash key matching (32-bit CRC) between the current block and reference blocks is extended to all allowed block sizes. The hash key calculation for each location in the current image is based on a 4×4 sub-block. For a larger current block, the hash key of the current block is considered to match the hash key of the reference block when the hash keys of all 4×4 sub-blocks match the hash keys at the corresponding reference locations. If multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matching reference is calculated, and the matching reference with the lowest cost is selected.

[0148] In block matching search, the search scope is set to cover both the previous and current CTUs.

[0149] At the CU level, IBC mode can be signaled using a flag, and can be signaled as IBC AMVP mode or IBC skip / merge mode, as shown below: - IBC Skip / Merge Mode: Uses a merge candidate index to indicate which block vector from the list of neighboring candidate IBC coding blocks to use to predict the current block. The merge candidate list consists of a space, HMVP, and paired candidates.

[0150] - IBC AMVP Mode: Block vector differences are encoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors, one from the left neighboring block and the other from the upper neighboring block (if IBC encoded). When neither neighboring is available, a default block vector is used as the predictor. A signaling flag identifies the index of the block vector predictor.

[0151] In current ECM designs, intra-block copy with local illumination compensation (IBC-LIC) is an encoding tool that compensates for local illumination variations within the image between the IBC-coded CU and its reference block using a linear model. The parameters of this linear model are derived using the same method as those for inter-frame prediction LIC, except that the reference template is generated using block vectors from the IBC.

[0152] IBC-LIC can be applied to both the IBC AMVP mode and the IBC merge mode. In the IBC AMVP mode, the IBC-LIC flag is signaled to indicate the use of IBC-LIC. In the IBC merge mode, the IBC-LIC flag is inherited from the encoded blocks based on the merge candidate index.

[0153] In some publicly available embodiments, three additional modes can be added to IBC-LIC to further improve coding performance. The first two modes are related to template shape selection. It is proposed to allow IBC-LIC to use only a top template, only a left template, or both top and left templates to derive individual model parameters. A multi-model IBC-LIC is also proposed, which allows IBC-LIC to have two linear models in a single CU. Both the top and left templates are used to derive the multi-model IBC-LIC parameters.

[0154] In the IBC AMVP mode, if the IBC flag is "true", an IBC-LIC index can be transmitted via signaling to indicate which IBC-LIC method is being used, as shown in Table 3.

[0155] Table 3: IBC-LIC Signaling Next, we describe the IBC filter. In ECM, a 7-tap filter consists of a 5-tap cross-shaped spatial component, a bias term, and a nonlinear term. For example... Figure 16As shown, the input of the spatial 5-tap component of the filter consists of the center (C) sample in the reference block and its upper / north (N), lower / south (S), left / west (W), and right / east (E) adjacent samples. The center (C) sample is located at the position corresponding to the sample in the current block to be predicted, as shown below.

[0156] The nonlinear term P is characterized as the square of the central sample C, and scaled to the range of sample values ​​of the content: P = (C) C + midVal )>>bitDepth (14) The bias term B represents the scalar offset between the input and the output and is set to an intermediate brightness value (512 for 10-bit content).

[0157] The output of the filter is calculated as follows: predLumaVal = c0C + c1N + c2S + c3E + c4W + c5P + c6B (15) The filter coefficient ci is calculated by minimizing the MSE between the reference template and the current template.

[0158] An additional indicator flag is introduced for the IBC-filtering, and this flag is transmitted in signaling depending on the IBC-LIC flag. Specifically, when the IBC-LIC flag is "true", this flag is transmitted in signaling and is used to indicate whether the IBC-filtering is applied to the current block. For IBC merging modes, this filtering mode is inherited when the merging mode list is generated, so no additional signaling is required.

[0159] In ECM, Combined Intra-Block Copy and Intra-Prediction (IBC-CIIP) is a coding tool for the CU that uses IBC and intra-prediction to obtain two prediction signals, which are then weighted and summed to generate the final prediction result, as follows: (16) in, and These represent the IBC prediction signal and the intra-frame prediction signal, respectively. For IBC combining mode and IBCAMVP mode, It was set to equal (13, 4) and (1, 1).

[0160] The intra-prediction signal is generated using an intra-prediction mode (IPM) candidate list, with the size of the IPM candidate list predefined to 2. An IPM index is transmitted via signaling to indicate which IPM to use.

[0161] In ECM, Intra-Block Copy with Geometric Partitioning (IBC-GPM) is a coding tool that geometrically partitions the CU into two sub-partitions. The predicted signals for these two sub-partitions are generated using IBC and intra-prediction. IBC-GPM can be applied to either regular IBC merging mode or IBC TM merging mode. The method for constructing the candidate list of Intra-Prediction Modes (IPM) is the same as for GPM using inter-frame and intra-frame prediction, and the size of the IPM candidate list is predefined as 3. There are a total of 48 geometric partitioning modes, which are divided into two sets of geometric partitioning modes, as follows: Table 4: Geometric Partition Patterns in the First Geometric Partition Pattern Set of IBC-GPM Table 5: Geometric Partition Patterns in the Second Geometric Partition Pattern Set of IBC-GPM When using IBC-GPM, the signaling transmits an IBC-GPM geometry partitioning mode set flag to identify whether the first or second geometry partitioning mode set is selected, followed by the geometry partitioning mode index. An IBC-GPM intra-frame flag is transmitted to indicate whether intra-frame prediction is used for the first sub-partition. When intra-frame prediction is used for a sub-partition, an intra-frame prediction mode index is transmitted. When IBC is used for a sub-partition, a merge index is transmitted.

[0162] The bidirectional predictive IBC GPM uses different IBCs to generate prediction samples for two GPM partitions. In the bidirectional predictive IBC GPM, the core IBC GPM design (e.g., 48 GPM modes, IBC merge candidate list) is consistent with the core IBC GPM design of the unidirectional predictive mode IBC GPM in ECM. Here, the unidirectional predictive IBC GPM uses both IBC and intra-frame modes to generate prediction samples for each GPM partition.

[0163] Next, bidirectional prediction IBC is described. IBC BVP-merging, inspired by AMVP-merging, derives the two required BVs from IBC block vector prediction (BVP) and IBC merging. Unlike the AMVP-merging mode, the two distinct indices of the IBC BVP and the IBC merging candidate are transmitted from the encoder to the decoder.

[0164] Bidirectional predictive IBC merging utilizes two different indices to derive the two BVs from an existing IBC merging candidate list. These two indices are transmitted from the encoder signaling to the decoder. The target of the bidirectional predictive IBC merging is IBCMBVD and IBC regular merging modes. In the current ECM, IBC-LIC and IBC-filtering are not applied to IBC-CIIP, IBC-GPM, or bidirectional predictive IBC.

[0165] However, the traditional LIC method has some problems. Firstly, while the traditional LIC method can locally compensate for illumination variations between different images, it is not applicable to GPM. Therefore, GPM cannot benefit from LIC. Secondly, the traditional LIC method only derives a linear model for all samples in the current block, which may lead to low prediction accuracy. Thirdly, for bidirectional prediction blocks, if the LIC flag is "true," the traditional LIC method must be applied to both unidirectional prediction blocks, which lacks flexibility. Fourthly, IBC-LIC and IBC-filtering are not applied to IBC-CIIP, IBC-GPM, or bidirectional prediction IBC.

[0166] According to the disclosed embodiments, in order to solve one or more of the above-mentioned problems associated with the LIC method, the LIC method can be applied to the GPM coding unit.

[0167] Some embodiments of this disclosure provide a method for decoding a bitstream. Figure 17A This is a schematic diagram illustrating an exemplary method 1700 for decoding a bitstream according to some embodiments of the present disclosure. Figure 17A As shown, method 1700 may include steps 1702 to 1704, which may be performed by a decoder (e.g., Figure 1 Image / video decoder 144 in the middle, and Figure 3A The decoding process shown is consistent with the 300A decoder and... Figure 3B The decoding process shown is consistent with the 300B decoder, or Figure 4 The device 400 is implemented by one or more processors associated with it.

[0168] In step 1702, the decoder can receive a bit stream.

[0169] In step 1704, the decoder can use the encoded information of the bitstream to decode one or more images. Specifically, the decoder can determine whether to apply a Local Illumination Compensation (LIC) procedure to the current block of the current image in the one or more images.

[0170] Figure 17B This illustrates some embodiments according to the present disclosure. Figure 17A The flowchart shows the sub-steps of method 1700. (See attached flowchart.) Figure 17B As shown, step 1704 may include sub-steps 1710 and 1712 that can be implemented by the decoder.

[0171] In substep 1710, the decoder can determine whether the current block of the current image is encoded by dividing it into multiple parts. In some embodiments, the current block can be divided according to a geometric partitioning pattern (GPM) by dividing the current block into two GPM partitions. In some embodiments, the current image can be encoded using an angle-weighted prediction (AWP) pattern. Although some embodiments of this disclosure are described in conjunction with GPM, it should be understood that such methods can also be applied to AWP.

[0172] In sub-step 1712, when it is determined in sub-step 1710 that the current block is divided into multiple parts, the multiple parts are predicted inter-frame based on their respective corresponding reference blocks. The decoder can determine whether to apply the LIC process to the multiple parts based on whether the encoded blocks in the current image have applied the Local Illumination Compensation (LIC) process.

[0173] In some embodiments, when the encoding unit encodes in GPM mode, the LIC flag can be inherited from the encoded blocks of each GPM partition portion. Specifically, in substep 1712, the decoder can determine whether to apply the LIC process to the target GPM partition portion of the two GPM partition portions based on whether the causal neighboring blocks of the current block have applied the LIC process. In some embodiments, the causal neighboring blocks can be blocks corresponding to the merge index of the target GPM partition portion.

[0174] In some embodiments, for a first GPM partition portion, a first LIC flag can be inherited from a first encoded block (e.g., a causal adjacent block) in the current image corresponding to a first merge index. Similarly, for a second GPM partition portion, a second LIC flag can be inherited from a second encoded block (e.g., a causal adjacent block) in the current image corresponding to a second merge index. Then, for each portion, if the inherited LIC flag is "true", the LIC procedure is applied.

[0175] Further reference Figure 17B In some embodiments, step 1704 further includes sub-step 1714, which may also be implemented by the decoder. In sub-step 1714, when it is determined that the LIC process will be applied to the target GPM partition, the decoder may determine the LIC parameters of the target GPM partition.

[0176] In some embodiments, the LIC parameter of the target GPM partition is determined based on the current template of the current block and the reference template of the reference block. For example, as Figure 18A As shown, the LIC process may include: based on the current template and the reference template (e.g., Figure 18A (using the first or second reference template shown), derive LIC parameters; and based on the LIC parameters, adjust the current portion after Motion Compensation (MC) (e.g., Figure 18A The predicted value of the first or second part (as shown in the diagram). When deriving the LIC parameters, the template can be constructed based on the neighboring samples available on the left and above. Finally, the predicted values ​​of the two parts adjusted by the LIC are fused to generate the final predicted value of the current coding unit.

[0177] In some embodiments, if the GPM partition portion is unidirectionally predicted and the inherited LIC flag is "true", the LIC method described above for unidirectionally predicted blocks can be used. In some embodiments, if the GPM partition portion is bidirectionally predicted and the inherited LIC flag is "true", the LIC method described above for bidirectionally predicted blocks can be used.

[0178] In some embodiments, step 1704 further includes sub-step 1716, which may also be implemented by the decoder. In sub-step 1716, when it is determined that the LIC process will be applied to the target GPM partition, and the target GPM partition is bidirectionally predicted, the decoder may skip decode-end motion vector correction (DMVR) or bidirectional optical flow (BDOF) for the target GPM partition. For example, if the GPM partition is bidirectionally predicted and the inherited LIC flag is "true", then DMVR (including multi-round DMVR, adaptive DMVR, and affine DMVR) and BDOF are not applied. The decoder may skip these processes.

[0179] In some embodiments, if the GPM partition is bidirectional prediction and meets the DMVR or BDOF conditions, the LIC flag of that partition can be set to "false" (e.g., 0), which means that LIC is not applied.

[0180] In some embodiments, if the GPM partition portion is bidirectionally predicted, the LIC flag for that portion can be set to "false" (e.g., 0). That is, the LIC method is not applied to bidirectionally predicted GPM partition portions.

[0181] In some embodiments, the LIC method can only be applied to GPM encoded blocks of low-latency images.

[0182] In some embodiments, when exporting LIC parameters, each GPM partition portion can be templated based on its respective GPM partition angle. As shown in Table 1, there are a total of 64 partitioning patterns, including 20 angles and 4 distances. For example, as... Figure 18B As shown, only the top template is used for exporting the LIC parameters in the first part, and only the left template is used for exporting the LIC parameters in the second part.

[0183] In some embodiments, when deriving the LIC parameters for the GPM partition portion, a template is first selected from the left adjacent sample, the top adjacent sample, or both the left and top adjacent samples, based on the partition angle. The basic principle for template selection is: if the partition has only a top adjacent sample, then the top adjacent sample is selected as the template; otherwise, if the partition has only a left adjacent sample, then the left adjacent sample is selected as the template; if neither of these conditions applies, then both the left and top adjacent samples are selected as the template. Specifically, for partition angles 0, 2, 3, 13, 14, 16, 18, 19, 29, and 30, the top adjacent sample is selected as the template for the first part using the first motion prediction; while the left and top adjacent samples are selected as the template for the second part using the second motion prediction. For partition angles 4 and 20, the top adjacent sample is selected as the template for the first part, and the left adjacent sample is selected as the template for the second part. For partition angles 5, 8, 11, 21, 24, and 27, the left and top adjacent samples are selected as the template for the first part, and the left adjacent sample is selected as the template for the second partition. For partition angles 12 and 28, the left and top adjacent samples are selected as templates for the first and second parts.

[0184] In some embodiments, the inherited LIC flags of each GPM partition portion are considered during GPM TM and GPM partitioning pattern reordering processes. During GPM TM or GPM partitioning pattern reordering, after the MC of the reference template, if the inherited LIC flags are "true", the LIC procedure is applied to the reference template, and subsequently, the TM cost between the current template and the reference template is calculated. The LIC parameters are derived based on the current template and the reference template, and the reference template after MC is adjusted using these derived LIC parameters.

[0185] In some embodiments, LIC parameters can be stored for the entire CU. For example, for a GPM-encoded CU, if the LIC applies only to one GPM partition, the LIC parameters for that partition are stored for the entire CU; if the LIC applies to two GPM partitions, based on the GPM partitioning pattern, only one set of LIC parameters from the two sets of LIC parameters for those two partitions is selected, and that set of parameters is stored for the entire CU. For example, the LIC parameters for the larger partition of the CU are selected. The stored LIC parameters can be used for the OBMC of adjacent CUs.

[0186] In some embodiments, for a GPM-encoded CU, if a LIC is applied, the LIC parameters are stored at the sub-block level. For example, the sub-block size can be 4×4 or 1×1. If the LIC is applied to two GPM partitions, which LIC parameter set is stored for a location in the CU depends on which partition the location belongs to. The stored LIC parameters can be used for the OBMC of adjacent CUs.

[0187] In some embodiments, the LIC is applied to GPM intra-inter mode coded blocks. For the current block, the LIC flag can be inherited from the coded blocks in the current image based on the merge index. If the LIC flag is "true", the LIC is applied to the inter-prediction block in GPM intra-inter mode. The LIC-adjusted inter-prediction block is then fused with the intra-prediction block to generate the final prediction result.

[0188] As mentioned above, the various methods of applying the LIC to GPM can also be applied to the AWP mode in the AVS3 standard.

[0189] According to some disclosed embodiments, a multi-model LIC method can be used to address one or more of the aforementioned problems associated with the LIC method.

[0190] In some embodiments, when the LIC method is applied to an encoding unit, two LIC models (two sets of LIC parameters) can be derived and applied. A threshold is determined. For example, the threshold can be the average of multiple samples in a reference template after MC. Alternatively, the threshold can be the average of multiple predicted samples in the current block after MC. The samples in the reference template are categorized into two classes based on the sample values ​​after MC and the threshold. For a sample in the reference template, if the value after MC is less than (or equal to) the threshold, the sample belongs to the first class; otherwise, the sample belongs to the second class. In some embodiments, two LIC models can be derived based on samples in the two classes respectively. For the current block, the samples are also categorized according to the same rules based on the predicted values ​​after MC and the threshold. For a sample in the current block, if the predicted value after MC is less than (or equal to) the threshold, the sample belongs to the first class, and the first LIC model is used to adjust the predicted value; otherwise, the sample belongs to the second class, and the second LIC model is used to adjust the predicted value. For example, in Figure 19 In this process, the blue samples in the template are used to derive the first LIC model, which is then applied to the blue samples in the current block to adjust the predicted values; while the green samples in the template are used to derive the second LIC model, which is then applied to the green samples in the current block.

[0191] In some embodiments, the multi-model approach can be used for bidirectional prediction of LIC. In one example, the two unidirectional prediction blocks in a bidirectional prediction block should have the same number of LIC models. For example, both unidirectional prediction blocks are tuned using multi-model LIC. This can be determined by the flags shown in Table 6. In one example, the two unidirectional prediction blocks in a bidirectional prediction block can have different numbers of LIC models. For example, one unidirectional prediction block is tuned using single-model LIC, while the other unidirectional prediction block is tuned using multi-model LIC. This can be determined by the indexes shown in Table 7. In some embodiments, the multi-model approach cannot be used for bidirectional prediction of LIC.

[0192] Table 6: Number of LIC models with two predictions in a bidirectional prediction LIC in an example.

[0193] Table 7: Number of LIC models for the two prediction blocks in a bidirectional prediction LIC in another example.

[0194] In some embodiments, the multi-model method can be used for coding units whose size (width × height) is greater than a certain threshold. For example, the threshold can be equal to 32, 64, 128, 256, or 512.

[0195] In some embodiments, for an AMVP mode coded block, if the LIC flag is "true", a second signaling flag is used to identify whether a single-model LIC or a multi-model LIC is applied. For a merged mode coded block, if the inherited LIC flag is "true", the second flag can also be inherited from the already coded block to determine whether a single-model LIC or a multi-model LIC is applied. In some embodiments, for a merged mode coded block, if the inherited LIC flag is "true", the second flag cannot be inherited from the already coded block, meaning that only single-model LIC can be applied to the merged mode coded block.

[0196] In some embodiments, if the LIC flag is "true", the choice between using single-model LIC and multi-model LIC is determined based on the template. For example, both single-model LIC and multi-model LIC are applied to the template, and then the method with the lower TM cost (i.e., the cost between the current template and the reference template after applying both LIC methods) is selected for the current block.

[0197] According to the disclosed embodiments, in order to address one or more of the aforementioned problems associated with the LIC method, the dual LIC method can be improved.

[0198] In the current ECM, for AMVP bidirectional prediction CUs, a signaling LIC flag is used to indicate whether LIC is applied to two unidirectional prediction blocks. For merged bidirectional prediction CUs, the LIC flag is inherited from the coded blocks to determine whether LIC is applied to two unidirectional prediction blocks.

[0199] In some embodiments, for a bidirectional prediction CU, the LIC may be applied to only one of the two unidirectional prediction blocks. For an AMVP bidirectional prediction CU, if the LIC flag is true, an index is further signaled to indicate whether the LIC is applied to both unidirectional prediction blocks or only one of the two unidirectional prediction blocks, as shown in Table 8. For a merged bidirectional prediction CU, if the inherited LIC flag is true, the index is also inherited from the encoded block to determine whether the LIC is applied to both unidirectional prediction blocks or only one of the two unidirectional prediction blocks.

[0200] Table 8: Number of LIC models for the two prediction blocks in a bidirectional prediction LIC in another example.

[0201] According to the disclosed embodiments, in order to address one or more of the aforementioned problems associated with the LIC method, IBC-LIC and IBC-filtering can be improved as follows.

[0202] Specifically, this disclosure proposes to apply IBC-LIC and / or IBC-filtering to IBC-CIIP, IBC-GPM, and bidirectional predictive IBC.

[0203] In some embodiments, the IBC-LIC is applied to the IBC-CIIP mode. If the IBC portion of the IBC-CIIP is encoded using the AMVP mode, a signaling flag is sent to indicate whether the IBC-LIC is applied to the IBC prediction block. If the IBC portion of the IBC-CIIP is encoded using the merge mode, the IBC-LIC flag can be inherited from the encoded blocks in the current image based on the merge index. If the IBC-LIC flag is "true", the IBC-LIC-adjusted IBC prediction block and the intra-prediction block are weighted and summed to generate the final prediction result. In some embodiments, when the IBC portion is encoded using the merge mode, the IBC-LIC is applied only to the IBC-CIIP mode.

[0204] In some embodiments, the IBC-LIC is applied to the IBC-GPM mode. For the unidirectional IBC-GPM mode (IBC prediction + intra-frame prediction), the IBC-LIC flag can be inherited from the coded blocks in the current image based on the merge index. If the IBC-LIC flag is "true", the IBC-LIC is applied to the IBC prediction. Then, the IBC prediction block adjusted by IBC-LIC is fused with the intra-frame prediction block to generate the final prediction result. For the bidirectional IBC-GPM mode, two LIC flags are inherited from the coded blocks in the current image based on the two merge indices of the two parts respectively. For each part, if the LIC flag is "true", the LIC is applied to the prediction of that part. Then, the prediction results of the two parts are fused to generate the final prediction result.

[0205] In some embodiments, the IBC-LIC is applied to bidirectional prediction of the IBC merging pattern. Two LIC flags are inherited from the encoded blocks in the current image, based on the two merging indices respectively. For each unidirectional prediction, if the LIC flag is "true", the LIC is applied to that prediction. The two unidirectional predictions are then weighted and summed to generate the final prediction result.

[0206] In some embodiments, if the inherited IBC-LIC flag is "true", the single-model IBC-LIC method using the top and left templates is applied to the current block. In some embodiments, if the inherited IBC-LIC flag is "true", an IBC-LIC index is also inherited, indicating which of the four IBC-LIC methods shown in Table 3. Furthermore, the IBC-LIC method corresponding to the inherited IBC-LIC index is applied to the current block.

[0207] In some embodiments, the IBC-filtering flag can be inherited in the same manner as described above in IBC-CIIP, IBC-GPM, and bidirectional predictive IBC merging modes.

[0208] It should be noted that the embodiments described in this disclosure can be freely combined.

[0209] In some embodiments, a method for encoding a video sequence into a bitstream is also provided. Figure 20A This is a schematic diagram illustrating an exemplary method 2000 for encoding a video sequence into a bitstream according to some embodiments of the present disclosure. Figure 20A As shown, method 2000 may include steps 2002 to 2006, which may be performed by an encoder (e.g., Figure 1 Image / video encoder 124 in, and Figure 2A The encoding process shown is consistent with the encoder of 200A, and... Figure 2B The encoding process shown is consistent with the 200B encoder, or Figure 4 The device 400 is implemented by one or more processors associated with it.

[0210] In step 2002, the encoder can receive a video sequence.

[0211] In step 2004, the encoder may encode one or more images of the video sequence. Specifically, the encoder may determine whether to apply a Local Illumination Compensation (LIC) process to the current block of the current image in the one or more images.

[0212] In step 2006, the encoder may generate a bitstream associated with the encoded image. The bitstream may include the encoded result generated in step 2004.

[0213] Figure 20B This illustrates some embodiments according to the present disclosure. Figure 20A A schematic diagram of a sub-step of method 2000 shown. In some embodiments, encoding one or more images of the video sequence in step 2004 may include, for example... Figure 20BThe following sub-steps 2010 to 2012 are shown, and these sub-steps can be implemented by the encoder.

[0214] In sub-step 2010, the encoder may determine whether the current block of the current image is encoded by dividing it into multiple parts. In some embodiments, the current image may be encoded using an Angle-Weighted Prediction (AWP) mode. While some embodiments of this disclosure are described in conjunction with GPM, it should be understood that such methods can also be applied to AWP.

[0215] In sub-step 2012, when it is determined in sub-step 2010 that the current block is divided into multiple parts, and the multiple parts are predicted inter-frame based on their respective reference blocks, the encoder can determine whether to apply the LIC process to the multiple parts based on whether the encoded blocks in the current image have applied the Local Illumination Compensation (LIC) process.

[0216] It is understandable that other aspects of Method 2000 can be inherited from the aforementioned decoding methods when applicable. For the sake of brevity, such methods will not be elaborated upon here.

[0217] In some embodiments, a decoding device includes: The receiving module is configured to receive bit streams; and A decoding module configured to decode one or more images using the encoded information of the bitstream. The decoding module is configured as follows: Determine whether the current block of the current image is encoded by dividing it into multiple parts; and When the current block is divided into multiple parts, it is determined whether the LIC process should be applied to the multiple parts based on whether the local illumination compensation (LIC) process has been applied to the encoded block.

[0218] In some embodiments, the current block is divided according to a geometric partitioning pattern (GPM) by dividing the current block into two GPM partitions.

[0219] In some embodiments, the decoding module is configured to: Based on whether the LIC process has been applied to the causal neighboring blocks of the current block, determine whether the LIC process should be applied to the target GPM partition in the two GPM partitions.

[0220] In some embodiments, the causal neighbor block is a block corresponding to the merged index of the target GPM partition.

[0221] In some embodiments, the decoding module is configured to: When it is determined that the LIC process will be applied to the target GPM partition, the LIC parameters for the target GPM partition are determined.

[0222] In some embodiments, the LIC parameter of the target GPM partition is determined based on the current template of the current block and the reference template of the reference block.

[0223] In some embodiments, the current template includes the left adjacent sample and the top adjacent sample of the current block, while the reference template includes the left adjacent sample and the top adjacent sample of the reference block.

[0224] In some embodiments, the current template is the left neighboring sample of the current block, and the reference template is the left neighboring sample of the reference block; The current template is the sample adjacent to the top of the current block, and the reference template is the sample adjacent to the top of the reference block; or The current template includes the left adjacent sample and the top adjacent sample of the current block, and the reference template includes the left adjacent sample and the top adjacent sample of the reference block.

[0225] In some embodiments, the decoding device further includes: A storage module configured to store the LIC parameters of the target GPM partition for a sub-block comprising the target GPM partition.

[0226] In some embodiments, the target GPM segmentation is predicted unidirectionally or bidirectionally.

[0227] In some embodiments, the decoding device further includes: A skip module is configured to skip decoder-end motion vector correction (DMVR) or bidirectional optical flow (BDOF) for the target GPM partition when it is determined that the LIC process is applied to the target GPM partition and the target GPM partition is bidirectionally predicted.

[0228] In some embodiments, the decoding module is configured to: When it is determined that the target GPM segment is bidirectionally predicted, and when decoder-side motion vector correction (DMVR) or bidirectional optical flow (BDOF) is applied to the target GPM segment, it is determined that the LIC process will not be applied to the target GPM segment.

[0229] In some embodiments, the decoding module is configured to: When it is determined that the target GPM partition is predicted bidirectionally, it is determined that the LIC process will not be applied to the target GPM partition.

[0230] In some embodiments, the decoding device further includes: An inheritance module is configured to inherit LIC flags from the causal neighbor blocks when it is determined that the LIC process will be applied to the target GPM partition, for use in GPM template matching (TM) or GPM partition pattern reordering for the target GPM partition.

[0231] In some embodiments, the decoding module is configured to: When it is determined that the LIC procedure will be applied to the target GPM partition in the two GPM partitions: Determine the LIC parameters of the target GPM partition; and For the current block, store the LIC parameters of the target GPM partition; or When it is determined that the LIC process will be applied to each of the two GPM partitioning parts: Determine the LIC parameter of one of the two GPM partitions; and For the current block, store the LIC parameter of one of the two GPM partitions.

[0232] In some embodiments, the current block is a coded block using GPM intra-inter-frame mode.

[0233] In some embodiments, the current image is encoded using an angle-weighted prediction (AWP) mode.

[0234] In some embodiments, an encoding device includes: A receiving module configured to receive video sequences; An encoding module configured to encode one or more images of the video sequence; and The generation module is configured to generate a bitstream. The encoding module is configured as follows: Determine whether the current block of the current image is encoded by dividing it into multiple parts; and When the current block is divided into multiple parts for inter-frame prediction based on the corresponding reference block, it is determined whether the LIC process should be applied to the multiple parts based on whether the coded block has applied the Local Illumination Compensation (LIC) process.

[0235] In some embodiments, the current block is divided according to a geometric partitioning pattern (GPM) by dividing the current block into two GPM partitions.

[0236] In some embodiments, a non-transitory computer-readable storage medium is also provided, on which a bitstream is stored. The bitstream can be encoded and decoded according to the disclosed local illumination compensation method.

[0237] In some embodiments, a non-transitory computer-readable storage medium including instructions is also provided, and the instructions can be executed by a device (e.g., the disclosed encoder and decoder) to perform the methods described above. Common forms of non-transitory media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs or any other flash memory, NVRAMs, caches, registers, any other memory chips or cassette tapes, and their networking versions. The device may include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memory.

[0238] In some embodiments, a non-transitory computer-readable storage medium is also provided, on which a video bitstream is stored. When the bitstream is decoded by a processor, the processor performs a method for decoding the bitstream to output one or more images of the video stream according to the method described above for decoding the bitstream to output one or more images of the video stream.

[0239] In some embodiments, a non-transitory computer-readable storage medium is also provided, on which a video bitstream is stored. When the bitstream is encoded by a processor, the processor executes multiple methods of encoding the video sequence into a bitstream according to the method described above.

[0240] In some embodiments, a computer program product is provided, the computer program product comprising: computer program instructions that enable a computer to perform a method for decoding a bitstream to output one or more images of a video stream, according to an embodiment of the method described above for decoding a bitstream to output one or more images of a video stream.

[0241] In some embodiments, a computer program product is provided, the computer program product comprising: computer program instructions that enable a computer to execute multiple methods of the method for encoding a video sequence into a bitstream according to the method embodiments of the method for encoding a video sequence into a bitstream described above.

[0242] In some embodiments, a computer program is provided that enables a computer to perform a method for decoding a bitstream to output one or more images of a video stream, according to the above-described method embodiments for decoding a bitstream to output one or more images of a video stream.

[0243] In some embodiments, a computer program is provided that enables a computer to execute multiple methods of the method for encoding a video sequence into a bitstream, according to the method embodiments of the method for encoding a video sequence into a bitstream described above.

[0244] The embodiments may be further described using the following terms: 1. A method for decoding a bitstream to output one or more images of a video stream, the method comprising: Receive bitstream; and Using the encoded information of the bitstream, one or more images are decoded. The decoding includes: Determine whether the current block of the current image is encoded by dividing it into multiple parts; and When the current block is divided into multiple parts, it is determined whether to apply the LIC process to the multiple parts based on whether the encoded block has applied the Local Illumination Compensation (LIC) process.

[0245] 2. The method according to Clause 1, wherein the current block is partitioned according to a geometric partitioning pattern (GPM) by dividing the current block into two GPM partitions.

[0246] 3. The method according to Clause 2, wherein determining whether to apply the LIC procedure to the plurality of parts includes: Based on whether the LIC process has been applied to the causal neighboring blocks of the current block, determine whether the LIC process should be applied to the target GPM partition in the two GPM partitions.

[0247] 4. The method according to Clause 3, wherein the causal adjacent block is a block corresponding to the merged index of the target GPM partition portion.

[0248] 5. The method described in Clause 3 further includes: When it is determined that the LIC process will be applied to the target GPM partition, the LIC parameters for the target GPM partition are determined.

[0249] 6. The method according to Clause 5, wherein the LIC parameter of the target GPM partition is determined based on the current template of the current block and the reference template of the reference block.

[0250] 7. The method according to Clause 6, wherein the current template includes the left adjacent sample and the top adjacent sample of the current block, and the reference template includes the left adjacent sample and the top adjacent sample of the reference block.

[0251] 8. The method according to Clause 6, wherein the current template is the left adjacent sample of the current block, and the reference template is the left adjacent sample of the reference block; The current template is the sample above the current block, and the reference template is the sample above the reference block; or The current template includes the left adjacent sample and the upper adjacent sample of the current block, while the reference template includes the left adjacent sample and the upper adjacent sample of the reference block.

[0252] 9. The method described in Clause 5 further includes: For each sub-block that includes the target GPM partition, the LIC parameters of the target GPM partition are stored.

[0253] 10. The method according to any one of clauses 3 to 9, wherein the target GPM segmentation is predicted unidirectionally or bidirectionally.

[0254] 11. The method according to any one of clauses 3 to 10, further comprising: When it is determined that the LIC process will be applied to the target GPM partition and the target GPM partition is bidirectionally predicted, the decoder-side motion vector correction (DMVR) or bidirectional optical flow (BDOF) for the target GPM partition is skipped.

[0255] 12. The method according to any one of clauses 3 to 11, wherein determining whether to apply the LIC procedure to the plurality of parts comprises: When it is determined that the target GPM segment is bidirectionally predicted, and decoder motion vector correction (DMVR) or bidirectional optical flow (BDOF) is applied to the target GPM segment, it is determined that the LIC process will not be applied to the target GPM segment.

[0256] 13. The method according to any one of clauses 3 to 12, wherein determining whether to apply the LIC procedure to the plurality of parts comprises: When it is determined that the target GPM partition is predicted bidirectionally, it is determined that the LIC process will not be applied to the target GPM partition.

[0257] 14. The method according to any one of clauses 3 to 13, further comprising: When it is determined that the LIC process will be applied to the target GPM partition, the LIC flag is inherited from the causal neighbor block for use in GPM template matching TM or GPM partition pattern reordering for the target GPM partition.

[0258] 15. The method according to any one of clauses 2 to 14, further comprising: When it is determined that the LIC procedure will be applied to the target GPM partition in the two GPM partitions: Determine the LIC parameters of the target GPM partition; and For the current block, store the LIC parameters of the target GPM partition; or When it is determined that the LIC process will be applied to each of the two GPM partitioning parts: Determine the LIC parameter of one of the two GPM partitions; and For the current block, store the LIC parameter of one of the two GPM partitions.

[0259] 16. The method according to any one of clauses 2 to 15, wherein the current block is a coded block in GPM intra-inter-frame mode.

[0260] 17. The method according to any one of clauses 1 to 16, wherein the current image is encoded using an angle-weighted prediction (AWP) mode.

[0261] 18. A method for encoding a video sequence into a bitstream, the method comprising: Receive video sequences; Encode one or more images of the video sequence; and Generate bitstream, The encoding includes: Determine whether the current block of the current image is encoded by dividing it into multiple parts; and When the current block is divided into multiple parts for inter-frame prediction based on the corresponding reference block, it is determined whether the LIC process should be applied to the multiple parts based on whether the coded block has applied the Local Illumination Compensation (LIC) process.

[0262] 19. The method according to Clause 18, wherein the current block is partitioned according to a geometric partitioning pattern (GPM) by dividing the current block into two GPM partitions.

[0263] 20. The method according to Clause 19, wherein determining whether to apply the LIC procedure to the plurality of parts includes: Based on whether the LIC process has been applied to the causal neighboring blocks of the current block, determine whether the LIC process should be applied to the target GPM partition in the two GPM partitions.

[0264] 21. The method according to Clause 20, wherein the causal adjacent block is a block corresponding to the merged index of the target GPM partition portion.

[0265] 22. The method described in accordance with Clause 20 further includes: When it is determined that the LIC process will be applied to the target GPM partition, the LIC parameters for the target GPM partition are determined.

[0266] 23. The method according to Clause 22, wherein the LIC parameter of the target GPM partition is determined based on the current template of the current block and the reference template of the reference block.

[0267] 24. The method according to Clause 23, wherein the current template includes the left adjacent sample and the top adjacent sample of the current block, and the reference template includes the left adjacent sample and the top adjacent sample of the reference block.

[0268] 25. The method according to Clause 23, wherein the current template is the left adjacent sample of the current block, and the reference template is the left adjacent sample of the reference block; The current template is the sample above the current block, and the reference template is the sample above the reference block; or The current template includes the left adjacent sample and the upper adjacent sample of the current block, while the reference template includes the left adjacent sample and the upper adjacent sample of the reference block.

[0269] 26. The method described in accordance with Clause 22 further includes: For each sub-block that includes the target GPM partition, the LIC parameters of the target GPM partition are stored.

[0270] 27. The method according to any one of clauses 20 to 26, wherein the target GPM segmentation is predicted unidirectionally or bidirectionally.

[0271] 28. The method according to any one of clauses 20 to 27, further comprising: When it is determined that the LIC process will be applied to the target GPM partition and the target GPM partition is bidirectionally predicted, the decoder-side motion vector correction (DMVR) or bidirectional optical flow (BDOF) for the target GPM partition is skipped.

[0272] 29. The method according to any one of clauses 20 to 28, wherein determining whether to apply the LIC procedure to the plurality of parts comprises: When it is determined that the target GPM segment is bidirectionally predicted, and decoder motion vector correction (DMVR) or bidirectional optical flow (BDOF) is applied to the target GPM segment, it is determined that the LIC process will not be applied to the target GPM segment.

[0273] 30. The method according to any one of clauses 20 to 29, wherein determining whether to apply the LIC procedure to the plurality of parts comprises: When it is determined that the target GPM partition is predicted bidirectionally, it is determined that the LIC process will not be applied to the target GPM partition.

[0274] 31. The method according to any one of clauses 20 to 30, further comprising: When it is determined that the LIC process will be applied to the target GPM partition, the LIC flag is inherited from the causal neighbor block for use in GPM template matching TM or GPM partition pattern reordering for the target GPM partition.

[0275] 32. The method according to any one of clauses 19 to 31, further comprising: When it is determined that the LIC procedure will be applied to the target GPM partition in the two GPM partitions: Determine the LIC parameters of the target GPM partition; and For the current block, store the LIC parameters of the target GPM partition; or When it is determined that the LIC process will be applied to each of the two GPM partitioning parts: Determine the LIC parameter of one of the two GPM partitions; and For the current block, store the LIC parameter of one of the two GPM partitions.

[0276] 33. The method according to any one of clauses 19 to 32, wherein the current block is a coded block in GPM intra-inter-frame mode.

[0277] 34. The method according to any one of clauses 18 to 33, wherein the current image is encoded using an angle-weighted prediction (AWP) mode.

[0278] 35. A decoding device, comprising: The receiving module is configured to receive bit streams; and A decoding module, configured to decode one or more images using the encoded information of the bitstream. The decoding module is configured as follows: Determine whether the current block of the current image is encoded by dividing it into multiple parts; and When the current block is divided into multiple parts, it is determined whether to apply the LIC process to the multiple parts based on whether the encoded block has applied the Local Illumination Compensation (LIC) process.

[0279] 36. The decoding apparatus according to clause 35, wherein the current block is divided according to a geometric partitioning pattern (GPM) by dividing the current block into two GPM partitions.

[0280] 37. The decoding apparatus according to clause 36, wherein the decoding module is configured to: Based on whether the LIC process has been applied to the causal neighboring blocks of the current block, determine whether the LIC process should be applied to the target GPM partition in the two GPM partitions.

[0281] 38. The decoding apparatus according to Clause 37, wherein the causal adjacent block is a block corresponding to the merged index of the target GPM partition portion.

[0282] 39. The decoding apparatus according to clause 37, wherein the decoding module is configured to: When it is determined that the LIC process will be applied to the target GPM partition, the LIC parameters for the target GPM partition are determined.

[0283] 40. The decoding apparatus according to Clause 39, wherein the LIC parameter of the target GPM partition is determined based on the current template of the current block and the reference template of the reference block.

[0284] 41. The decoding apparatus according to Clause 40, wherein the current template includes the left adjacent sample and the upper adjacent sample of the current block, and the reference template includes the left adjacent sample and the upper adjacent sample of the reference block.

[0285] 42. The decoding apparatus according to clause 40, wherein the current template is the left adjacent sample of the current block, and the reference template is the left adjacent sample of the reference block; The current template is the sample above the current block, and the reference template is the sample above the reference block; or The current template includes the left adjacent sample and the upper adjacent sample of the current block, while the reference template includes the left adjacent sample and the upper adjacent sample of the reference block.

[0286] 43. The decoding apparatus according to clause 39, further comprising: A storage module configured to store the LIC parameters of the target GPM partition for a sub-block comprising the target GPM partition.

[0287] 44. The decoding apparatus according to any one of clauses 37 to 43, wherein the target GPM partition is predicted unidirectionally or bidirectionally.

[0288] 45. The decoding apparatus according to any one of clauses 37 to 44, further comprising: A skip module is configured to skip decoder-end motion vector correction (DMVR) or bidirectional optical flow (BDOF) for the target GPM partition when it is determined that the LIC process is applied to the target GPM partition and the target GPM partition is bidirectionally predicted.

[0289] 46. ​​The decoding apparatus according to any one of clauses 37 to 45, wherein the decoding module is configured to: When it is determined that the target GPM segment is bidirectionally predicted, and decoder motion vector correction (DMVR) or bidirectional optical flow (BDOF) is applied to the target GPM segment, it is determined that the LIC process will not be applied to the target GPM segment.

[0290] 47. The decoding apparatus according to any one of clauses 37 to 46, wherein the decoding module is configured to: When it is determined that the target GPM partition is predicted bidirectionally, it is determined that the LIC process will not be applied to the target GPM partition.

[0291] 48. The decoding apparatus according to any one of clauses 37 to 47, further comprising: An inheritance module is configured to inherit LIC flags from causal adjacent blocks when it is determined that the LIC process will be applied to the target GPM partition, for use in GPM template matching TM or GPM partition pattern reordering for the target GPM partition.

[0292] 49. The decoding apparatus according to any one of clauses 36 to 48, wherein the decoding module is configured to: When it is determined that the LIC procedure will be applied to the target GPM partition in the two GPM partitions: Determine the LIC parameters of the target GPM partition; and For the current block, store the LIC parameters of the target GPM partition; or When it is determined that the LIC process will be applied to each of the two GPM partitioning parts: Determine the LIC parameter of one of the two GPM partitions; and For the current block, store the LIC parameter of one of the two GPM partitions.

[0293] 50. The decoding apparatus according to any one of clauses 36 to 49, wherein the current block is an encoded block in GPM intra-inter-frame mode.

[0294] 51. The decoding apparatus according to any one of clauses 35 to 50, wherein the current image is encoded in an angle-weighted prediction (AWP) mode.

[0295] 52. An encoding device, the encoding device comprising: A receiving module configured to receive video sequences; An encoding module configured to encode one or more images of the video sequence; and The generation module is configured to generate a bitstream. The encoding module is configured as follows: Determine whether the current block of the current image is encoded by dividing it into multiple parts; and When the current block is divided into multiple parts for inter-frame prediction based on the corresponding reference block, it is determined whether the LIC process should be applied to the multiple parts based on whether the coded block has applied the Local Illumination Compensation (LIC) process.

[0296] 53. The encoding apparatus according to Clause 52, wherein the current block is divided according to a geometric partitioning pattern (GPM) by dividing the current block into two GPM partitions.

[0297] 54. An electronic device, comprising: A memory storing an instruction set; and one or more processors configured to execute the instruction set to cause the one or more processors to perform a method of decoding a bitstream to output one or more images of a video stream in accordance with any one of clauses 1 to 17.

[0298] 55. An electronic device comprising: A memory that stores an instruction set; and one or more processors configured to execute the instruction set to cause the one or more processors to perform a method of encoding a video sequence into a bitstream according to any one of clauses 18 to 34.

[0299] 56. A non-transitory computer-readable storage medium having a video bitstream stored thereon for processing according to a method comprising: Determine whether the current block was encoded by dividing it into multiple parts; and When the current block is divided into multiple parts for inter-frame prediction based on the corresponding reference block, it is determined whether the LIC process should be applied to the multiple parts based on whether the coded block has applied the Local Illumination Compensation (LIC) process.

[0300] 57. The non-transitory computer-readable storage medium as described in Clause 56, wherein the current block is partitioned according to a geometric partitioning pattern (GPM), that is, the current block is divided into two GPM partitions.

[0301] 58. The non-transitory computer-readable storage medium as described in Clause 57, wherein determining whether to apply the LIC procedure to the plurality of parts includes: Based on whether the LIC process has been applied to the causal neighboring blocks of the current block, determine whether the LIC process should be applied to the target GPM partition in the two GPM partitions.

[0302] 59. The non-transitory computer-readable storage medium as described in Clause 58, wherein the causal adjacent blocks are blocks corresponding to the merged index of the target GPM partition portion.

[0303] 60. The non-transitory computer-readable storage medium according to Clause 58, wherein the method further comprises: When it is determined that the LIC process will be applied to the target GPM partition, the LIC parameters for the target GPM partition are determined.

[0304] 61. The non-transitory computer-readable storage medium according to Clause 60, wherein the LIC parameter of the target GPM partition is determined based on the current template of the current block and the reference template of the reference block.

[0305] 62. The non-transitory computer-readable storage medium according to Clause 61, wherein the current template includes the left adjacent sample and the upper adjacent sample of the current block, and the reference template includes the left adjacent sample and the upper adjacent sample of the reference block.

[0306] 63. The non-transitory computer-readable storage medium according to Clause 61, wherein the current template is the left adjacent sample of the current block, and the reference template is the left adjacent sample of the reference block; The current template is the sample above the current block, and the reference template is the sample above the reference block; or The current template includes the left adjacent sample and the upper adjacent sample of the current block, while the reference template includes the left adjacent sample and the upper adjacent sample of the reference block.

[0307] 64. The non-transitory computer-readable storage medium according to clause 60, wherein the method further comprises: For each sub-block that includes the target GPM partition, the LIC parameters of the target GPM partition are stored.

[0308] 65. A non-transitory computer-readable storage medium according to any one of clauses 58 to 64, wherein the target GPM partition portion is predicted unidirectionally or bidirectionally.

[0309] 66. The non-transitory computer-readable storage medium according to any one of clauses 58 to 65, wherein the method further comprises: When it is determined that the LIC process will be applied to the target GPM partition and the target GPM partition is bidirectionally predicted, the decoder-side motion vector correction (DMVR) or bidirectional optical flow (BDOF) for the target GPM partition is skipped.

[0310] 67. A non-transitory computer-readable storage medium according to any one of clauses 58 to 66, wherein determining whether to apply the LIC procedure to the plurality of parts includes: When it is determined that the target GPM segment is bidirectionally predicted, and decoder motion vector correction (DMVR) or bidirectional optical flow (BDOF) is applied to the target GPM segment, it is determined that the LIC process will not be applied to the target GPM segment.

[0311] 68. A non-transitory computer-readable storage medium according to any one of clauses 58 to 67, wherein determining whether to apply the LIC procedure to the plurality of parts includes: When it is determined that the target GPM partition is predicted bidirectionally, it is determined that the LIC process will not be applied to the target GPM partition.

[0312] 69. The non-transitory computer-readable storage medium according to any one of clauses 58 to 68, wherein the method further comprises: When it is determined that the LIC process will be applied to the target GPM partition, the LIC flag is inherited from the causal neighbor block for use in GPM template matching TM or GPM partition pattern reordering for the target GPM partition.

[0313] 70. The non-transitory computer-readable storage medium according to any one of clauses 57 to 69, wherein the method further comprises: When it is determined that the LIC procedure will be applied to the target GPM partition in the two GPM partitions: Determine the LIC parameters of the target GPM partition; and For the current block, store the LIC parameters of the target GPM partition; or When it is determined that the LIC process will be applied to each of the two GPM partitioning parts: Determine the LIC parameter of one of the two GPM partitions; and For the current block, store the LIC parameter of one of the two GPM partitions.

[0314] 71. A non-transitory computer-readable storage medium according to any one of clauses 57 to 70, wherein the current block is an encoded block in GPM intra-inter-frame mode.

[0315] 72. The non-transitory computer-readable storage medium according to any one of clauses 56 to 72, wherein the current image is encoded in an angle-weighted prediction (AWP) mode.

[0316] 73. A non-transitory computer-readable storage medium having a video sequence of a video stored thereon, wherein, when the video sequence of the video is encoded by a processor, the processor performs a method for encoding the video sequence into a bit stream according to any one of claims 18 to 34.

[0317] 74. A computer program product comprising computer program instructions, wherein the computer program instructions enable a computer to perform a method for decoding a bitstream to output one or more images of a video stream according to any one of clauses 1 to 17.

[0318] 75. A computer program product comprising computer program instructions, wherein the computer program instructions enable a computer to perform a method for encoding a video sequence into a bitstream according to any one of clauses 18 to 34.

[0319] 76. A computer program, wherein the computer program enables a computer to perform a method for decoding a bitstream to output one or more images of a video stream according to any one of the provisions 1 to 17.

[0320] 77. A computer program, wherein the computer program enables a computer to perform a method for encoding a video sequence into a bitstream according to any one of claims 18 to 34.

[0321] It should be noted that the relational terms such as "first," "second," etc., used in this document are only used to distinguish one entity or operation from another, and do not require or imply any actual relationship or order between these entities or operations. Furthermore, the words "comprising," "having," "containing," and "including," as well as other similar forms, are intended to be identical in meaning and are open-ended, because one or more items following any of these words do not imply an exhaustive list of such one or more items, nor do they imply limitation to only the listed one or more items.

[0322] As used herein, unless otherwise specified, the term "or" covers all possible combinations unless impractical. For example, if it is specified that a database may include A or B, then unless otherwise specified or impractical, the database may include A, or B, or A and B. As a second example, if it is specified that a database may include A, B, or C, then unless otherwise specified or impractical, the database may include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0323] It should be understood that the above embodiments can be implemented in hardware, software (program code), or a combination of hardware and software. If implemented in software, it can be stored in the aforementioned computer-readable medium. When executed by a processor, the software can perform the disclosed methods. The computing units and other functional units described in this disclosure can be implemented in hardware, software, or a combination of hardware and software. Those skilled in the art should also understand that multiple modules / units described above can be combined into one module / unit, and each module / unit described above can be further divided into multiple sub-modules / sub-units.

[0324] In the foregoing specification, numerous specific details have been described with reference to embodiments, which may vary depending on the implementation. Certain adjustments and modifications may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art upon consideration of the specification and practice of this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims. The sequence of steps shown in the figures is also considered for illustrative purposes only and is not intended to limit one to any particular sequence of steps. Therefore, those skilled in the art will understand that these steps may be performed in a different order while implementing the same method.

[0325] Exemplary embodiments have been disclosed in the accompanying drawings and description. However, many variations and modifications can be made to these embodiments. Therefore, although specific terminology has been used, it is used only in a general and descriptive sense and not for limiting purposes.

Claims

1. A method for decoding a bitstream to output one or more images of a video stream, the method comprising: Receive bit stream; as well as Using the encoded information of the bitstream, one or more images are decoded. The decoding includes: Determine whether the current block of the current image is encoded by dividing it into multiple parts; and When the current block is divided into multiple parts, it is determined whether to apply the LIC process to the multiple parts based on whether the encoded block has applied the Local Illumination Compensation (LIC) process.

2. The method according to claim 1, wherein, The current block is divided into two GPM partitions, and the current block is divided according to the geometric partitioning pattern GPM.

3. The method according to claim 2, wherein, Determining whether to apply the LIC procedure to the plurality of parts includes: Based on whether the LIC process has been applied to the causal neighboring blocks of the current block, determine whether the LIC process should be applied to the target GPM partition in the two GPM partitions.

4. The method according to claim 3, wherein, The causal adjacent block is a block corresponding to the merged index of the target GPM partition.

5. The method of claim 3, further comprising: When it is determined that the LIC process will be applied to the target GPM partition, the LIC parameters for the target GPM partition are determined.

6. The method according to claim 5, wherein, The LIC parameters of the target GPM partition are determined based on the current template of the current block and the reference template of the reference block.

7. The method according to claim 6, wherein, The current template includes the left adjacent sample and the top adjacent sample of the current block, while the reference template includes the left adjacent sample and the top adjacent sample of the reference block.

8. The method according to claim 6, wherein, The current template is the left adjacent sample of the current block, and the reference template is the left adjacent sample of the reference block; The current template is the sample above the current block, and the reference template is the sample above the reference block; or The current template includes the left adjacent sample and the upper adjacent sample of the current block, while the reference template includes the left adjacent sample and the upper adjacent sample of the reference block.

9. The method of claim 5, further comprising: For each sub-block that includes the target GPM partition, the LIC parameters of the target GPM partition are stored.

10. The method according to any one of claims 3 to 9, wherein, The target GPM segmentation is predicted unidirectionally or bidirectionally.

11. The method according to any one of claims 3 to 10, further comprising: When it is determined that the LIC process is applied to the target GPM partition and the target GPM partition is bidirectionally predicted, the decoder motion vector correction (DMVR) or bidirectional optical flow (BDOF) for the target GPM partition is skipped.

12. The method according to any one of claims 3 to 11, wherein, Determining whether to apply the LIC procedure to the plurality of parts includes: When it is determined that the target GPM segment is bidirectionally predicted, and decoder motion vector correction (DMVR) or bidirectional optical flow (BDOF) is applied to the target GPM segment, it is determined that the LIC process will not be applied to the target GPM segment.

13. The method according to any one of claims 3 to 12, wherein, Determining whether to apply the LIC procedure to the plurality of parts includes: When it is determined that the target GPM partition is predicted bidirectionally, it is determined that the LIC process will not be applied to the target GPM partition.

14. The method according to any one of claims 3 to 13, further comprising: When it is determined that the LIC process will be applied to the target GPM partition, the LIC flag is inherited from the causal neighbor block for use in GPM template matching TM or GPM partition pattern reordering for the target GPM partition.

15. The method according to any one of claims 2 to 14, further comprising: When it is determined that the LIC procedure will be applied to the target GPM partition in the two GPM partitions: Determine the LIC parameters of the target GPM partition; and For the current block, store the LIC parameters of the target GPM partition; or When it is determined that the LIC process will be applied to each of the two GPM partitioning parts: Determine the LIC parameter of one of the two GPM partitions; and For the current block, store the LIC parameter of one of the two GPM partitions.

16. The method according to any one of claims 2 to 15, wherein, The current block is a coded block using GPM intra-inter frame mode.

17. The method according to any one of claims 1 to 16, wherein, The current image is encoded using the Angle-Weighted Prediction (AWP) mode.

18. A method for encoding a video sequence into a bitstream, the method comprising: Receive video sequences; Encode one or more images from the video sequence; as well as Generate bitstream, The encoding includes: Determine whether the current block of the current image is encoded by dividing it into multiple parts; and When the current block is divided into multiple parts for inter-frame prediction based on the corresponding reference block, it is determined whether the LIC process should be applied to the multiple parts based on whether the coded block has applied the Local Illumination Compensation (LIC) process.

19. The method according to claim 18, wherein, The current block is divided into two GPM partitions, and the current block is partitioned according to the GPM partitioning mode.

20. A non-transitory computer-readable storage medium having a video bitstream stored thereon for processing according to a method comprising: Determine whether the current block was encoded by dividing it into multiple parts; as well as When the current block is divided into multiple parts for inter-frame prediction based on the corresponding reference block, it is determined whether the LIC process should be applied to the multiple parts based on whether the coded block has applied the Local Illumination Compensation (LIC) process.

21. A computer program product comprising computer program instructions, wherein, The computer program instructions enable the computer to perform a method for decoding a bitstream to output one or more images of a video stream according to any one of claims 1 to 17.

22. A computer program product comprising computer program instructions, wherein, The computer program instructions enable the computer to perform the method of encoding a video sequence into a bitstream as described in claim 18 or 19.

23. A computer program, wherein, The computer program enables a computer to perform a method for decoding a bitstream to output one or more images of a video stream according to any one of claims 1 to 17.

24. A computer program, wherein, The computer program enables the computer to perform the method of encoding a video sequence into a bitstream as described in claim 18 or 19.

Images