Method for building a merge candidate list

Abstract

The present application provides a system and method for constructing a merge candidate list for video processing. An exemplary method includes inserting a set of spatial merge candidates into a merge candidate list for a coding block, wherein the set of spatial merge candidates is inserted in the following order: upper neighboring block, left neighboring block, upper neighboring block, left neighboring block, and upper-left neighboring block. The method may also include adding at least one of the following to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 62 / 902,790, filed on September 19, 2019, the entire contents of which are incorporated herein by reference. Background Art

[0003] A video is a set of static pictures (or "frames") that capture visual information. In order to reduce storage memory and transmission bandwidth, the video can be compressed before storage or transmission, and decompressed before display. The compression process is usually called encoding, and the decompression process is usually called decoding. There are currently a variety of video coding formats that use standardized video coding technologies. The most common ones are video coding formats based on prediction, transform, quantization, entropy coding, and in-loop filtering. Video coding standards that specify specific video coding formats, such as the High Efficiency Video Coding (HEVC / H.265) standard, the Versatile Video Coding (VVC / H.266) standard, the AVS standard, etc., are developed by standardization organizations. As more and more advanced video coding technologies are adopted in video standards, the coding efficiency of new video coding standards is also getting higher and higher. Summary of the Invention

[0004] Embodiments of the present application provide a method for constructing a merge candidate list. According to some embodiments, an exemplary method includes inserting a set of spatial merge candidates into a merge candidate list for a coding block, wherein the spatial merge candidates are inserted in the following order: upper neighboring block, left neighboring block, upper neighboring block, left neighboring block, and upper-left neighboring block.

[0005] According to some embodiments, an exemplary method includes inserting a set of spatial merge candidates into a merge candidate list for a coding block based on a preset number limit. If the number limit is 2, the spatial merge candidates are inserted into the merge candidate list in the following order: upper neighboring block, left neighboring block. If the number limit is 3, the spatial merge candidates are inserted into the merge candidate list in the following order: upper neighboring block, left neighboring block, upper neighboring block.

[0006] According to some embodiments, an exemplary method includes: inserting a set of spatial merging candidates into a merge candidate list of a coding block, wherein: when a first coding mode is applied to the coding block, the spatial merging candidates are inserted in a first construction order; when a second coding mode is applied to the coding block, the spatial merging candidates are inserted in a second construction order, and the first construction order is different from the second construction order.

[0007] According to some embodiments, an exemplary method includes: inserting a set of spatial merging candidates into a merge candidate list of a coding block, wherein: when the coding block belongs to a low-latency picture, the spatial merging candidates are inserted in a first construction order; when the coding block belongs to a non-low-latency picture, the spatial merging candidates are inserted in a second construction order, and the first construction order is different from the second construction order. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The following detailed description and accompanying drawings illustrate embodiments and aspects of the present application. The various features shown in the drawings are not drawn to scale.

[0009] Figure 1 The structure of an example video sequence is shown according to some embodiments of the present application.

[0010] Figure 2A A schematic diagram showing an exemplary encoding process of a hybrid video coding system consistent with an embodiment of the present application is shown.

[0011] Figure 2B A schematic diagram showing another exemplary encoding process of a hybrid video coding system consistent with an embodiment of the present application is shown.

[0012] Figure 3A A schematic diagram showing an exemplary decoding process of a hybrid video coding system consistent with an embodiment of the present application is shown.

[0013] Figure 3B A schematic diagram showing another exemplary decoding process of a hybrid video coding system consistent with an embodiment of the present application is shown.

[0014] Figure 4A A block diagram illustrating an exemplary apparatus for encoding or decoding video consistent with embodiments of the present application is shown.

[0015] Figure 4B Exemplary locations of spatial merging candidates consistent with embodiments of the present application are shown.

[0016] Figure 4C Exemplary positions of temporal merging candidates consistent with embodiments of the present application are shown.

[0017] Figure 5 An exemplary scaling of temporal merging candidates consistent with embodiments of the present application is shown.

[0018] Figure 6 An exemplary relationship between the distance index and the predefined offset and the motion vector difference (MMVD) in the merge mode consistent with an embodiment of the present application is shown.

[0019] Figure 7Exemplary locations of spatial merging candidates consistent with embodiments of the present application are shown.

[0020] Figure 8 Exemplary experimental results consistent with embodiments of the present application compared with VTM-6 in a random access (RA) configuration are shown.

[0021] Figure 9 Exemplary experimental results consistent with embodiments of the present application are shown compared to VTM-6 in a low-latency (LD) configuration.

[0022] Figure 10 Exemplary experimental results consistent with embodiments of the present application are shown compared to VTM-6 in an RA configuration.

[0023] Figure 11 Exemplary experimental results consistent with embodiments of the present application are shown compared to VTM-6 in an LD configuration.

[0024] Figure 12 An exemplary syntax structure of a fragment header consistent with an embodiment of the present application is shown.

[0025] Figure 13 An exemplary syntax structure of a sequence parameter set (SPS) consistent with an embodiment of the present application is shown.

[0026] Figure 14 An exemplary syntax structure of a picture parameter set (PPS) consistent with an embodiment of the present application is shown.

[0027] Figure 15 Exemplary experimental results consistent with embodiments of the present application are shown compared to VTM-6 in an RA configuration.

[0028] Figure 16 Exemplary experimental results consistent with embodiments of the present application are shown compared to VTM-6 in an LD configuration.

[0029] Figure 17 Exemplary experimental results consistent with embodiments of the present application are shown compared to VTM-6 in an RA configuration.

[0030] Figure 18 Exemplary experimental results consistent with embodiments of the present application are shown compared to VTM-6 in an LD configuration.

[0031] Figure 19 A flowchart of an exemplary video processing method consistent with an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] Reference can now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings. The following description is made with reference to the accompanying drawings, in which the same numbers in different figures represent the same or similar elements, unless otherwise specified. The embodiments set forth in the following description of example embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with the aspects related to the present application described in the appended claims. Certain aspects of the present application are described in more detail below. In the event of a conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall prevail.

[0033] Videos are frames arranged in a temporal sequence to store visual information. A video capture device (e.g., a camera) can be used to capture and store these images in temporal order, and a video playback device (e.g., a television, computer, smartphone, tablet, video player, or any end-user terminal with a display) can be used to display these images in temporal order. Furthermore, in some applications, the video capture device can transmit the captured video to a video playback device (e.g., a computer with a monitor) in real time, for example, for monitoring, conferencing, or live broadcasting.

[0034] To reduce the storage space and transmission bandwidth required for such applications, video can be compressed before storage and transmission, and decompressed before display. Compression and decompression can be implemented by software executed by a processor (e.g., a general-purpose computer processor) or dedicated hardware. The module used for compression is generally referred to as an "encoder," and the module used for decompression is generally referred to as a "decoder." Encoders and decoders can be collectively referred to as "codecs." Encoders and decoders can be implemented as any of a variety of suitable hardware, software, or combinations thereof. For example, hardware implementations of encoders and decoders can include circuits such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, or any combination thereof. Software implementations of encoders and decoders can include program code, computer-executable instructions, firmware, or any suitable computer-implemented algorithm or process fixed in a computer-readable medium. Video compression and decompression can be implemented using various algorithms or standards, such as MPEG-1, MPEG-2, MPEG-4, the H.26x series, etc. In some applications, a codec can decompress video from a first coding standard and recompress the decompressed video using a second coding standard. In this case, the codec can be referred to as a "transcoder."

[0035] The video encoding process identifies and retains useful information that can be used to reconstruct the image, while ignoring unimportant information during the reconstruction process. If the ignored, unimportant information cannot be fully reconstructed, the encoding process is called "lossy." Otherwise, it is called "lossless." Most encoding processes are lossy, a trade-off made to reduce required storage space and transmission bandwidth.

[0036] Useful information about the picture being encoded (called the "current picture") includes changes relative to a reference picture (e.g., a previously encoded and reconstructed picture). Such changes can include changes in pixel position, brightness, or color, with position changes being of greatest interest. A change in the position of a group of pixels representing an object can reflect the object's motion between the reference picture and the current picture.

[0037] A picture that is encoded without reference to another picture (i.e., it is its own reference picture) is called an "I-picture." A picture that is encoded using a previous picture as a reference picture is called a "P-picture." A picture that is encoded using both a previous picture and a future picture as reference pictures (i.e., the reference is "bidirectional") is called a "B-picture."

[0038] To achieve the same subjective quality as HEVC / H.265 using half the bandwidth, JVET has been developing technologies beyond HEVC using the Joint Exploration Model (JEM) reference software. As coding technologies are incorporated into JEM, JEM achieves higher coding performance than HEVC.

[0039] The VVC standard further includes more coding technologies, which provide better compression performance. VVC is based on the same hybrid video coding system used in modern video compression standards (such as HEVC, H.264 / AVC, MPEG2, H.263, etc.). In VVC, a merge candidate list containing new merge candidates can be constructed. For different inter-frame modes, merge lists of different sizes are applied. The embodiments of the present application take into account new merge candidates in VVC (such as HMVP, paired average) and new inter-frame modes (such as MMVD, TPM). For example, the order of spatial merge candidates can be improved, and the number of spatial merge candidates can be adjusted. In addition, for normal mode, MMVD and TPM mode, the structure of spatial merge candidates is fixed.

[0040] Figure 1The structure of an example video sequence 100 according to some embodiments of the present application is illustrated. The video sequence 100 can be real-time video or video that has been captured and archived. The video 100 can be real-life video, computer-generated video (e.g., computer game video), or a combination thereof (e.g., real-life video with augmented reality effects). The video sequence 100 can be input from a video capture device (e.g., a camera), a video archive containing previously captured video (e.g., a video file stored on a storage device), or a video input interface (e.g., a video broadcast transceiver) to receive video from a video content provider.

[0041] like Figure 1 As shown, video sequence 100 may include a series of pictures arranged temporally along a timeline, including pictures 102, 104, 106, and 108. Pictures 102-106 are consecutive, and there are more pictures between pictures 106 and 108. Figure 1 In the example, picture 102 is an I-picture, and its reference picture is picture 102 itself. Picture 104 is a P-picture, as indicated by the arrow, and its reference picture is picture 102. Picture 106 is a B-picture, as indicated by the arrow, and its reference pictures are pictures 104 and 108. In some embodiments, the reference pictures of a picture (e.g., picture 104) may not be directly before or after the picture. For example, the reference picture of picture 104 may be a picture before picture 102. It should be noted that the reference pictures of pictures 102-106 are only examples, and this application does not limit the embodiments of reference pictures to Figure 1 The example shown in .

[0042] Due to the computational complexity of such tasks, video codecs typically do not encode or decode an entire picture at once. Instead, they can divide the picture into basic segments and encode or decode the picture segment by segment. In this application, these basic segments are called basic processing units ("BPUs"). For example, Figure 1Structure 110 in shows an example structure of a picture (e.g., any of pictures 102-108) of video sequence 100. In structure 110, the picture is divided into 4×4 basic processing units, whose boundaries are shown as dashed lines. In some embodiments, the basic processing unit may be referred to as a "macroblock" in some video coding standards (e.g., the MPEG family, H.261, H.263, or H.264 / AVC), or as a "coding tree unit" ("CTU") in some other video coding standards (e.g., H.265 / HEVC or H.266 / VVC). The basic processing unit in a picture can have different sizes, such as 128×128, 64×64, 32×32, 16×16, 4×8, 16×32, or pixels of arbitrary shape and size. The size and shape of the basic processing unit for a picture can be selected based on a balance between coding efficiency and the level of detail to be retained in the basic processing unit.

[0043] A basic processing unit may be a logical unit that may include a set of different types of video data stored in a computer memory (e.g., in a video frame buffer). For example, a basic processing unit for a color picture may include a luma component (Y) representing achromatic luma information, one or more chroma components (e.g., Cb and Cr) representing color information, and associated syntax elements, where the luma and chroma components may have basic processing units of the same size. In some video coding standards (e.g., H.265 / HEVC or H.266 / VVC), the luma and chroma components may be referred to as "coding tree blocks" ("CTBs"). Any operation performed on a basic processing unit may be repeated for each of its luma and chroma components.

[0044] Video encoding has multiple stages of operation, examples of which are given in Figures 2A-2B and Figures 3A-3BAs shown in . For each stage, the size of the basic processing unit may still be too large to be processed, so it can be further divided into segments referred to as "basic processing sub-units" in this application. In some embodiments, the basic processing sub-unit may be referred to as a "block" in some video coding standards (for example, the MPEG series, H.261, H.263 or H.264 / AVC), or as a "coding unit" ("CU") in certain other video coding standards (for example, H.265 / HEVC or H.266 / VVC). The basic processing sub-unit may have the same or smaller size as the basic processing unit. Similar to the basic processing unit, the basic processing sub-unit is also a logical unit, which may include storage in computer memory (for example, in a video frame buffer). Any operation performed on the basic processing sub-unit can be repeated for each of its luminance and chrominance components. It should be noted that this division can be performed to a further level depending on the processing requirements. It should also be noted that different stages can use different schemes to divide the basic processing units.

[0045] For example, in the mode decision phase (an example of which is Figure 2B ), the encoder can decide which prediction mode to use for a basic processing unit (e.g., intra-picture prediction or inter-picture prediction), and the basic processing unit may be too large to make such a decision. The encoder can split the basic processing unit into multiple basic processing sub-units (e.g., CUs in H.265 / HEVC or H.266 / VVC) and determine the prediction type for each individual basic processing sub-unit.

[0046] For another example, in the prediction phase (an example of which is Figures 2A-2B ), the encoder can perform prediction operations at the level of basic processing sub-units (e.g., CUs). However, in some cases, the basic processing sub-units may still be too large to process. The encoder can further split the basic processing sub-units into smaller segments (e.g., called "prediction blocks" or "PBs" in H.265 / HEVC or H.266 / VVC), at which level prediction operations can be performed.

[0047] For another example, in the transformation phase (an example of which is Figures 2A-2B), the encoder can perform transform operations on the residual basic processing sub-units (e.g., CUs). However, in some cases, these basic processing sub-units may still be too large to process. The encoder can further split the basic processing sub-units into smaller segments (e.g., called "transform blocks" or "TBs" in H.265 / HEVC or H.266 / VVC), and transform operations can be performed at the level of the segments. It should be noted that the division scheme of the same basic processing sub-unit can be different in the prediction stage and the transform stage. For example, in H.265 / HEVC or H.266 / VVC, the prediction blocks and transform blocks of the same CU can have different sizes and numbers.

[0048] exist Figure 1 In the structure 110, the basic processing unit 112 is further divided into 3×3 basic processing sub-units, whose boundaries are shown by dotted lines. In different solutions, different basic processing units of the same image can be divided into different basic processing sub-units.

[0049] In some embodiments, to provide parallel processing and fault resilience for video encoding and decoding, a picture can be divided into multiple processing regions. This allows the encoding or decoding process for a certain region of the picture to be independent of information from any other region of the picture. In other words, each region of the picture can be processed independently. In this way, the codec can process different regions of the picture in parallel, thereby improving coding efficiency. In addition, when data in one region is corrupted during processing or lost during network transmission, the codec can correctly encode or decode other regions of the same picture without relying on the damaged or lost data, thereby providing fault resilience. In some video coding standards, a picture can be divided into different types of regions. For example, H.265 / HEVC and H.266 / VVC provide two types of regions: "segments" and "tiles." It should also be noted that different pictures in the video sequence 100 can have different partitioning schemes for dividing the picture into regions.

[0050] For example, in Figure 1 In FIG, the structure 110 is divided into three regions 114, 116 and 118, whose boundaries are shown as solid lines inside the structure 110. Region 114 includes four basic processing units. Each of regions 116 and 118 includes six basic processing units. It should be noted that Figure 1 The basic processing units, basic processing sub-units, and regions of the structure 110 are merely examples, and the present application does not limit their implementation.

[0051] Figure 2A Schematic diagram of an example encoding process 200A consistent with an embodiment of the present invention is shown. For example, the encoding process 200A can be performed by an encoder. Figure 2A As shown, the encoder can encode the video sequence 202 into a video code stream 228 according to the encoding process 200A. Figure 1 The video sequence 100 in FIG. 200 may include a set of pictures (referred to as “original pictures”) arranged in time sequence. Figure 1 Based on the structure 110 in FIG. 1 , the encoder may divide each original picture of the video sequence 202 into basic processing units, basic processing sub-units, or regions for processing. In some embodiments, the encoder may perform encoding process 200A at the basic processing unit level for each original picture of the video sequence 202. For example, the encoder may perform encoding process 200A in an iterative manner, wherein the encoder may encode a basic processing unit in one iteration of process 200A. In some embodiments, the encoder may perform process 200A in parallel for regions (e.g., regions 114-118) of each original picture of the video sequence 202.

[0052] exist Figure 2A 200A, an encoder may input a basic processing unit (BPU) of an original picture of a video sequence 202 (referred to as an "original BPU") into a prediction stage 204 to generate prediction data 206 and a prediction BPU 208. The encoder may subtract the prediction BPU 208 from the original BPU to generate a residual BPU 210. The encoder may input the residual BPU 210 into a transform stage 212 and a quantization stage 214 to generate quantized transform coefficients 216. The encoder may input the prediction data 206 and the quantized transform coefficients 216 into a binary encoding stage 226 to generate a video code stream 228. Components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 may be referred to as a "forward path." During process 200A, after the quantization stage 214, the encoder may input the quantized transform coefficients 216 into an inverse quantization stage 218 and an inverse transform stage 220 to generate a reconstructed residual BPU 222. The encoder can add the reconstructed residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224, which is used in the prediction stage 204 for the next iteration of process 200A. Components 218, 220, 222, and 224 of process 200A can be referred to as a "reconstruction path." The reconstruction path can be used to ensure that both the encoder and decoder use the same reference data for prediction.

[0053] The encoder can iteratively perform process 200A to encode each original BPU of the original picture (in the forward path) and generate a prediction reference 224 for encoding the next original BPU of the original picture (in the reconstruction path). After encoding all the original BPUs of the original picture, the encoder can continue to encode the next picture in the video sequence 202.

[0054] Referring to process 200A, an encoder may receive a video sequence generated by a video capture device (e.g., a camera) 202. The term "receive" as used herein may refer to any action of receiving, inputting, acquiring, retrieving, obtaining, reading, accessing, or any manner for inputting data.

[0055] In the prediction stage 204, in the current iteration, the encoder may receive the original BPU and the prediction reference 224 and perform a prediction operation to generate the prediction data 206 and the predicted BPU 208. The prediction reference 224 may be generated from the reconstruction path of the previous iteration of the process 200A. The purpose of the prediction stage 204 is to reduce information redundancy by extracting the prediction data 206 that can be used to reconstruct the original BPU into the predicted BPU 208 from the prediction data 206 and the prediction reference 224.

[0056] Ideally, the predicted BPU 208 would be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 208 typically differs slightly from the original BPU. To account for this difference, after generating the predicted BPU 208, the encoder can subtract it from the original BPU to generate a residual BPU 210. For example, the encoder can subtract the value of the corresponding pixel of the original BPU from the value (e.g., grayscale value or RGB value) of the pixel corresponding to the predicted BPU 208. Each pixel of the residual BPU 210 can have a residual value that is the result of this subtraction between the corresponding pixel of the original BPU and the predicted BPU 208. Compared to the original BPU, the predicted data 206 and the residual BPU 210 can have fewer bits, but they can be used to reconstruct the original BPU without significantly reducing quality, thereby compressing the original BPU.

[0057] To further compress the residual BPU 210, in the transform stage 212, the encoder can reduce its spatial redundancy by decomposing the residual BPU 210 into a set of two-dimensional "basis patterns", each of which is associated with a "transform coefficient". The basis patterns can have the same size (e.g., the size of the residual BPU 210). Each basis pattern can represent a frequency-varying (e.g., frequency-varying) component of the residual BPU 210. No basis pattern can be reproduced from any combination (e.g., linear combination) of any other basis patterns. In other words, the decomposition can decompose the variation of the residual BPU 210 into the frequency domain. This decomposition is analogous to a discrete Fourier transform of a function, where the basis patterns are analogous to the basis functions (e.g., trigonometric functions) of the discrete Fourier transform, and the transform coefficients are analogous to the coefficients associated with the basis functions.

[0058] Different transform algorithms can use different base patterns. In the transform stage 212, various transform algorithms can be used, such as discrete cosine transform, discrete sine transform, etc. The transform in the transform stage 212 is reversible. That is, the encoder can restore the residual BPU 210 by performing the inverse operation of the transform (referred to as an "inverse transform"). For example, to restore the pixels of the residual BPU 210, the inverse transform may be multiplying the values ​​of the corresponding pixels of the base pattern by the corresponding correlation coefficients and adding the products to produce a weighted sum. For video coding standards, both the encoder and decoder can use the same transform algorithm (and therefore the same base pattern). Therefore, the encoder can only record the transform coefficients, and the decoder can reconstruct the residual BPU 210 from the transform coefficients without receiving the base pattern from the encoder. Compared to the residual BPU 210, the transform coefficients may have fewer bits, but they can be used to reconstruct the residual BPU 210 without significantly reducing quality. As a result, the residual BPU 210 is further compressed.

[0059] The encoder can further compress the transform coefficients during the quantization stage 214. During the transform process, different basis patterns can represent different frequencies of change (e.g., the frequency of brightness changes). Because the human eye is generally better at detecting low-frequency changes, the encoder can ignore information about high-frequency changes without significantly degrading decoding quality. For example, during the quantization stage 214, the encoder can generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (called a "quantization parameter") and rounding the quotient to the nearest integer. This operation can convert some transform coefficients of high-frequency basis patterns to zero, while transform coefficients of low-frequency basis patterns can be converted to smaller integers. The encoder can ignore zero-valued quantized transform coefficients 216, thereby further compressing the transform coefficients. The quantization process is also reversible, where the quantized transform coefficients 216 can be reconstructed into transform coefficients in the inverse operation of quantization (called "inverse quantization").

[0060] Because the encoder ignores the remainder of this division during rounding operations, the quantization stage 214 can be lossy. Generally, the quantization stage 214 can cause the greatest information loss in process 200A. The greater the information loss, the fewer bits may be required to quantize the transform coefficients 216. To achieve different levels of information loss, the encoder can use different values ​​for the quantization parameter or any other parameter in the quantization process.

[0061] In the binary encoding stage 226, the encoder may encode the prediction data 206 and the quantized transform coefficients 216 using a binary encoding technique (e.g., entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless or lossy compression algorithm). In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the encoder may encode other information in the binary encoding stage 226, such as the prediction mode used in the prediction stage 204, the parameters of the prediction operation, the transform type in the transform stage 212, the parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), etc. The encoder may use the output data of the binary encoding stage 226 to generate a video stream 228. In some embodiments, the video stream 228 may be further packaged for network transmission.

[0062] Referring to the reconstruction path of process 200A, at the inverse quantization stage 218, the encoder may perform inverse quantization on the quantized transform coefficients 216 to generate reconstructed transform coefficients. At the inverse transform stage 220, the encoder may generate a reconstructed residual BPU 222 based on the reconstructed transform coefficients. The encoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a prediction reference 224 to be used in the next iteration of process 200A.

[0063] It should be noted that other variations of process 200A may also be used to encode video sequence 202. In some embodiments, the encoder may perform the stages of process 200A in a different order. In some embodiments, one or more stages of process 200A may be combined into a single stage. In some embodiments, a single stage of process 200A may be divided into multiple stages. For example, transform stage 212 and quantization stage 214 may be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, process 200A may omit one or more stages shown in FIG. 2 .

[0064] Figure 2B A schematic diagram of another example encoding process 200B consistent with embodiments of the present application is shown. Process 200B can be modified from process 200A. For example, process 200B can be used by encoders compliant with hybrid video coding standards (e.g., the H.26x series). Compared to process 200A, the forward path of process 200B additionally includes a mode decision stage 230 and divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B additionally includes a loop filter stage 232 and a buffer 234.

[0065] Generally, prediction techniques can be divided into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-picture prediction or "intra-frame prediction") can use pixels from one or more coded adjacent BPUs in the same picture to predict the current BPU. That is, the prediction reference 224 in spatial prediction can include adjacent BPUs. Spatial prediction can reduce the spatial redundancy inherent in the picture. Temporal prediction (e.g., inter-picture prediction or "inter-frame prediction") can use regions from one or more coded pictures to predict the current BPU. That is, the prediction reference 224 in temporal prediction can include a coded picture. Temporal prediction can reduce the temporal redundancy inherent in the picture.

[0066] Referring to process 200B, in the forward path, the encoder performs prediction operations in a spatial prediction stage 2042 and a temporal prediction stage 2044. For example, in the spatial prediction stage 2042, the encoder may perform intra-frame prediction. For the original BPU of the picture being encoded, the prediction reference 224 may include one or more neighboring BPUs that have been encoded (in the forward path) and reconstructed (in the reconstruction path) in the same picture. The encoder may generate a predicted BPU 208 by extrapolating the neighboring BPUs. Extrapolation techniques may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, and the like. In some embodiments, the encoder may perform extrapolation at the pixel level, for example, by extrapolating the value of the corresponding pixel for each pixel of the predicted BPU 208. The neighboring BPUs used for extrapolation may be positioned relative to the original BPU in various directions, such as vertically (e.g., above the original BPU), horizontally (e.g., to the left of the original BPU), diagonally (e.g., below, below, above, or above the original BPU), or in any direction defined in the video coding standard being used. For intra prediction, the prediction data 206 may include, for example, the location (eg, coordinates) of the used neighboring BPUs, the size of the used neighboring BPUs, extrapolation parameters, the direction of the used neighboring BPUs relative to the original BPU, and the like.

[0067] For another example, during the temporal prediction stage 2044, the encoder may perform inter-frame prediction. For the original BPU of the current picture, the prediction reference 224 may include one or more pictures (referred to as "reference pictures") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference pictures may be encoded and reconstructed using the BPU. For example, the encoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a reconstructed BPU. When all reconstructed BPUs for the same picture are generated, the encoder may generate the reconstructed picture as the reference picture. The encoder may perform a "motion estimation" operation to search for a matching region within a range of reference pictures (referred to as a "search window"). The position of the search window in the reference picture may be determined based on the position of the original BPU in the current picture. For example, the search window may be centered in the reference picture at a location with the same coordinates as the original BPU in the current picture and may extend outward a predetermined distance. When the encoder identifies (e.g., using a pixel recursive algorithm, a block matching algorithm, etc.) a region similar to the original BPU in the search window, the encoder may determine such a region as a matching region. The matching region may have a different size (e.g., smaller, equal, larger, or different shape) than the original BPU. Figure 1 ), so it can be considered that the matching area "moves" to the position of the original BPU over time. The encoder can record the direction and distance of this movement as a "motion vector". Figure 1 When the encoder is performing a multi-image search of the reference picture 106), the encoder can search for matching areas and determine its associated motion vector for each reference picture. In some embodiments, the encoder can assign weights to the pixel values ​​of the matching areas of each matching reference picture.

[0068] Motion estimation can be used to identify various types of motion, such as translation, rotation, scaling, etc. For inter-frame prediction, the prediction data 206 may include, for example, the location (e.g., coordinates) of the matching region, the motion vector associated with the matching region, the number of reference pictures, the weights associated with the reference pictures, etc.

[0069] To generate the predicted BPU 208, the encoder may perform a "motion compensation" operation. Motion compensation may be used to reconstruct the predicted BPU 208 based on the prediction data 206 (e.g., motion vectors) and the prediction reference 224. For example, the encoder may move a matching area of ​​the reference picture according to the motion vector, where the encoder may predict the original BPU of the current picture. When multiple reference pictures are used (e.g., Figure 1For example, for a picture 106 in the reference picture, the encoder can move the matching region of the reference picture based on the corresponding motion vector and the average pixel value of the matching region. In some embodiments, if the encoder has assigned weights to the pixel values ​​of the matching region of each matching reference picture, the encoder can add the weighted sum of the pixel values ​​of the moved matching region.

[0070] In some embodiments, inter-frame prediction can be unidirectional or bidirectional. Unidirectional inter-frame prediction can use one or more reference pictures in the same temporal direction as the current picture. For example, Figure 1 The picture 104 in is a unidirectional inter-frame prediction picture, where the reference picture (i.e., picture 102) precedes the picture 104. Bidirectional inter-frame prediction can use one or more reference pictures in both temporal directions relative to the current picture. For example, Figure 1 Picture 106 in is a bidirectional inter-predicted picture, where the reference pictures (ie, pictures 104 and 108 ) are in both temporal directions relative to picture 104 .

[0071] Still referring to the forward path of process 200B, after the spatial prediction stage 2042 and the temporal prediction stage 2044, in the mode decision stage 230, the encoder can select a prediction mode (e.g., one of intra-frame prediction and inter-frame prediction) for the current iteration of process 200B. For example, the encoder can perform a rate-distortion optimization technique, in which the encoder can select a prediction mode based on the bit rate of a candidate prediction mode and the distortion of a reference picture reconstructed under the candidate prediction mode to minimize the value of a cost function. Based on the selected prediction mode, the encoder can generate a corresponding predicted BPU 208 and predicted data 206.

[0072] In the reconstruction path of process 200B, if intra prediction mode is selected in the forward path, after generating prediction reference 224 (e.g., the current BPU in the current picture that has been encoded and reconstructed), the encoder can directly input prediction reference 224 into spatial prediction stage 2042 for later use (e.g., for extrapolating the next BPU of the current picture). If inter prediction mode is selected in the forward path, after generating prediction reference 224 (e.g., the current picture in which all BPUs have been encoded and reconstructed), the encoder can input prediction reference 224 into loop filter stage 232, at which point the encoder can apply a loop filter to prediction reference 224 to reduce or eliminate distortion (e.g., blocking artifacts) introduced by inter prediction. The encoder can apply various loop filter techniques in loop filter stage 232, such as deblocking, sample adaptive offset, adaptive loop filter, etc. The loop filtered reference picture can be stored in buffer 234 (or "decoded picture buffer") for later use (e.g., as an inter prediction reference picture for a future picture in the video sequence 202). The encoder may store one or more reference pictures in a buffer 234 for use in a temporal prediction stage 2044. In some embodiments, the encoder may encode parameters of the loop filter (e.g., loop filter strength) in a binary encoding stage 226, along with the quantized transform coefficients 216, prediction data 206, and other information.

[0073] Figure 3A A schematic diagram of an example decoding process 300A consistent with an embodiment of the present application is shown. Process 300A may be a decompression process corresponding to compression process 200A in FIG. In some embodiments, process 300A may be similar to the reconstruction path of process 200A. The decoder may decode video code stream 228 into video stream 304 according to process 300A. Video stream 304 may be very similar to video sequence 202. However, due to information loss during compression and decompression (e.g., Figures 2A-2B quantization stage 214 in), typically, the video stream 304 is different from the video sequence 202. Figures 2A-2B 200A and 200B, the decoder may perform process 300A at the basic processing unit (BPU) level for each picture encoded in the video stream 228. For example, the decoder may perform process 300A in an iterative manner, where the decoder may decode a basic processing unit in one iteration of decoding process 300A. In some embodiments, the decoder may perform process 300A in parallel for each region (e.g., regions 114-118) of each picture encoded in the video stream 228.

[0074] In Figure A, the decoder may input a portion of the video codestream 228 associated with a basic processing unit (referred to as a "coded BPU") for a coded picture into a binary decoding stage 302. In the binary decoding stage 302, the decoder may decode the portion into prediction data 206 and quantized transform coefficients 216. The decoder may input the quantized transform coefficients 216 into an inverse quantization stage 218 and an inverse transform stage 220 to generate a reconstructed residual BPU 222. The decoder may input the prediction data 206 into the prediction stage 204 to generate a predicted BPU 208. The decoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a predicted reference 224. In some embodiments, the predicted reference 224 may be stored in a buffer (e.g., a decoded picture buffer in computer memory). The decoder may input the predicted reference 224 into the prediction stage 204 for performing a prediction operation in the next iteration of process 300A.

[0075] The decoder may iteratively perform process 300A to decode each coded BPU of a coded picture and generate a prediction reference 224 for encoding the next coded BPU of the coded picture. After decoding all coded BPUs of the coded picture, the decoder may output the picture to a video stream 304 for display and continue decoding the next coded picture in the video code stream 228.

[0076] In the binary decoding stage 302, the decoder may perform the inverse of the binary coding technique used by the encoder (e.g., entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the decoder may also decode other information in the binary decoding stage 302, such as the prediction mode, parameters of the prediction operation, the transform type, the quantization parameter process (e.g., quantization parameter), encoder control parameters (e.g., bit rate control parameter), etc. In some embodiments, if the video stream 228 is transmitted over the network in packet form, the decoder may depacketize the video stream 228 before inputting it into the binary decoding stage 302.

[0077] Figure 3B A schematic diagram of another example decoding process 300B consistent with embodiments of the present application is shown. Process 300B can be modified from process 300A. For example, process 300B can be used by decoders compliant with hybrid video coding standards (e.g., the H.26x series). Compared to process 300A, process 300B additionally divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and additionally includes a loop filter stage 232 and a buffer 234.

[0078] In process 300B, for a coded basic processing unit (referred to as a "current BPU") of a coded picture being decoded (referred to as a "current picture"), the prediction data 206 decoded by the decoder from the binary decoding stage 302 may include various types of data, depending on what prediction mode the encoder used to encode the current BPU. For example, if the encoder used intra-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating intra-frame prediction, parameters of the intra-frame prediction operation, etc. The parameters of the intra-frame prediction operation may include, for example, the location (e.g., coordinates) of one or more neighboring BPUs used as a reference, the size of the neighboring BPUs, extrapolation parameters, the direction of the neighboring BPU relative to the original BPU, etc. For example, if the encoder uses inter-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator indicating inter-frame prediction (e.g., a flag value), parameters of the inter-frame prediction operation, etc. The parameters of the inter-frame prediction operation may include, for example, the number of reference pictures associated with the current BPU, the weights associated with the reference pictures respectively, the positions (e.g., coordinates) of one or more matching regions in the corresponding reference pictures, one or more motion vectors associated with the matching regions respectively, etc.

[0079] Based on the prediction mode indicator, the decoder can decide whether to perform spatial prediction (e.g., intra prediction) in the spatial prediction stage 2042 or temporal prediction (e.g., inter prediction) in the temporal prediction stage 2044. The details of performing such spatial prediction or temporal prediction are described in Figure 2B After performing such spatial prediction or temporal prediction, the decoder can generate a predicted BPU 208. Figure 3A As described in , the decoder may add the predicted BPU 208 and the reconstructed residual BPU 222 to generate a prediction reference 224 .

[0080] In process 300B, the decoder may input the prediction reference 224 to the spatial prediction stage 2042 or the temporal prediction stage 2044 to perform a prediction operation in the next iteration of process 300B. For example, if the current BPU is decoded using intra-frame prediction in the spatial prediction stage 2042, then after generating the prediction reference 224 (e.g., the decoded current BPU), the decoder may input the prediction reference 224 directly to the spatial prediction stage 2042 for later use (e.g., for extrapolating the next BPU of the current picture). If the current BPU is decoded using inter-frame prediction in the temporal prediction stage 2044, then after generating the prediction reference 224 (e.g., the reference picture in which all BPUs have been decoded), the encoder may input the prediction reference 224 to the loop filter stage 232 to reduce or eliminate distortion (e.g., blocking artifacts). The decoder may Figure 2B The loop filter is applied to the prediction reference 224 in the manner described in

[15] . The loop-filtered reference picture may be stored in a buffer 234 (e.g., a decoded picture buffer in a computer memory) for later use (e.g., as an inter-frame prediction reference picture for a future coded picture in the video codestream 228). The decoder may store one or more reference pictures in the buffer 234 for use in the temporal prediction stage 2044. In some embodiments, when the prediction mode indicator of the prediction data 206 indicates that inter-frame prediction was used to encode the current BPU, the prediction data may further include parameters of the loop filter (e.g., loop filter strength).

[0081] Figure 4A 4 is a block diagram of an example apparatus 400 for encoding or decoding a video consistent with an embodiment of the present application. Figure 4A As shown, the device 400 may include a processor 402. When the processor 402 executes the instructions described herein, the device 400 may become a special-purpose machine for video encoding or decoding. The processor 402 may be any type of circuit capable of manipulating or processing information. For example, the processor 402 may include any number of central processing units (or "CPUs"), graphics processing units (or "GPUs"), neural processing units ("NPUs"), microcontroller units ("MCUs"), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLAs), programmable array logic (PALs), general array logic (GALs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), systems on chip (SoCs), application-specific integrated circuits (ASICs), and the like. In some embodiments, the processor 402 may also be a group of processors grouped into a single logic controller. For example, as Figure 4A As shown, processor 402 may include multiple processors, including processor 402a, processor 402b, and processor 402n.

[0082] The device 400 may also include a memory 404 configured to store data (eg, a set of instructions, computer code, intermediate data, etc.). Figure 4AAs shown, the stored data may include program instructions (e.g., program instructions for implementing stages in process 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequence 202, video code stream 228, or video stream 304). Processor 402 may access program instructions and data for processing (e.g., via bus 410) and execute program instructions to operate or manipulate the data for processing. Memory 404 may include a high-speed random access memory device or a non-volatile memory device. In some embodiments, memory 404 may include any number of random access memories (RAM), read-only memories (ROM), optical disks, magnetic disks, hard drives, solid-state drives, flash drives, secure digital (SD) cards, memory sticks, compact flash (CF) cards, etc. Memory 404 may also be a group of memories grouped into a single logical component ( Figure 4A not shown).

[0083] The bus 410 may be a communication device that transmits data between components inside the apparatus 400 , such as an internal bus (eg, a CPU-memory bus), an external bus (eg, a Universal Serial Bus port, a Peripheral Component Interconnect Express port), and the like.

[0084] For ease of explanation and to avoid ambiguity, the processor 402 and other data processing circuitry are collectively referred to as "data processing circuitry" in this application. The data processing circuitry may be implemented entirely as hardware, a combination of software and hardware, or firmware. Furthermore, the data processing circuitry may be a single standalone module, or may be fully or partially integrated into any other component of the apparatus 400.

[0085] The device 400 may also include a network interface 406 to provide wired or wireless communications with a network (e.g., the Internet, an intranet, a local area network, a mobile communication network, etc.). In some embodiments, the network interface 406 may include any number of network interface controllers (NICs), radio frequency (RF) modules, repeaters, transceivers, modems, routers, gateways, any combination of wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication ("NFC") adapters, cellular network chips, etc.

[0086] In some embodiments, the apparatus 400 may optionally further include a peripheral interface 408 to provide a connection to one or more peripheral devices. Figure 4A As shown, peripheral devices may include, but are not limited to, a cursor control device (e.g., a mouse, touchpad, or touch screen), a keyboard, a display (e.g., a cathode ray tube display, a liquid crystal display, or a light emitting diode display), a video input device (e.g., a camera or a communication input interface coupled to a video archive), etc.

[0087] It should be noted that a video codec (e.g., a codec that performs processes 200A, 200B, 300A, or 300B) can be implemented as any combination of software or hardware modules in apparatus 400. For example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more software modules of apparatus 400, such as program instructions that can be loaded into memory 404. For another example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more hardware modules of apparatus 400, such as dedicated data processing circuits (e.g., FPGAs, ASICs, NPUs, etc.).

[0088] For a CU coded using inter-frame prediction, a reference block in a previously decoded picture (i.e., a reference picture) is identified as a predictor. The relative position between the reference block in the reference picture and the coded block in the current picture is defined as a motion vector (MV). The motion information of the current CU is specified by the number of predictors, the reference picture index, and the corresponding MV. After the prediction is obtained through motion compensation based on the motion information, the residual between the predicted signal and the original signal can be additionally transformed, quantized, and entropy coded before being packaged into the output bitstream.

[0089] In some cases, the motion information of the current CU can be used to predict the motion information of the current CU using the motion information of its spatial and temporal neighboring CUs. A merge mode can be used to reduce the coding bits of the motion information. In the merge mode, the motion information is derived from spatial or temporal neighboring blocks, and a marked merge index can be used to indicate which neighboring block the motion information is derived from.

[0090] In HEVC, a merge candidate list can be constructed based on the following candidates:

[0091] (1) A maximum of four spatial merging candidates from five spatially adjacent blocks;

[0092] (2) a temporal merge candidate derived from the temporally co-located block;

[0093] (3) Additional merge candidates, including combined bi-directional prediction candidates and zero motion vector candidates.

[0094] The first candidate in the merge candidate list is the spatial neighbor. Figure 4BThe positions of five spatial candidates are shown. Each candidate position is checked for availability according to the order {A1, B1, B0, A0, B2}. If a spatial neighboring block is intra-predicted or the position is outside the current fragment or tile, it can be considered an unavailable merge candidate (merge candidate block). In addition, some redundancy checks can be performed to ensure that the motion data from the neighboring blocks are as unique as possible. In order to reduce the complexity caused by redundant checks, only limited redundant checks can be performed, and uniqueness may not always be guaranteed. For example, given the order {A1, B1, B0, A0, B2}, B0 only checks B1, A0 only checks A1, and B2 only checks A1 and B1.

[0095] like Figure 4C As shown in , for temporal merging candidates, the lower right position C0 outside the collocated block of the reference picture is used if available. Otherwise, the center position C1 can be used instead. The index marked in the slice header can be used to indicate which reference picture list is used for the collocated reference picture. Figure 5 As shown, the MVs of co-located blocks can be scaled according to the difference in picture order count (POC) before being inserted into the merge list.

[0096] The maximum number C of merge candidates can be specified in the segment header. If the number of available merge candidates found (including the temporal candidates) is greater than C, only the first C-1 spatial and temporal candidates are retained. Otherwise, if the number of available merge candidates is less than C, additional candidates are generated until the number equals C. Because the ability to parse the coded data does not depend on the number of available merge candidates, parsing can be simplified and made more robust. In the Common Test Condition (CTC), the maximum number C of merge candidates is set to 5.

[0097] For B slices, additional merge candidates are generated by combining two available candidates according to the predefined order of reference picture lists 0 and 1. For example, the first generated candidate uses the first merge candidate from list 0 and the second merge candidate from list 1. HEVC specifies a total of 12 predefined pairs of two motion vectors in the constructed merge candidate list. Their order is (0,1), (1,0), (0,2), (2,1), (0,3), (3,0), (1,3), (3,1), (2,3), (3,2), (3,2), (0,2), (0,3), (1,3), (3,1), (2,3), (3,2), where (i,j) represents the index of the available merge candidate. After removing redundant entries, a maximum of five candidates can be included.

[0098] When the slice is a P slice or the number of merge candidates is still less than C, zero motion vectors associated with reference indices (from 0 to the number of reference pictures minus 1) are used to fill any remaining entries in the merge candidate list.

[0099] In VVC, the merge candidate list is constructed in sequence from the following five candidates:

[0100] Spatial merging candidates from spatially neighboring CUs;

[0101] Temporal merging candidates from collocated CUs;

[0102] History-based motion vector predictor (HMVP) from FIFO table;

[0103] Pairwise average candidates; and

[0104] Zero MV.

[0105] The definitions of spatial merge candidates and temporal merge candidates are the same as those in HEVC. After the spatial and temporal merge candidates, the HMVP merge candidates are added to the merge list. In HMVP, the motion information of the previously coded blocks is stored in a table and used as the motion vector predictor of the current CU. The table containing multiple HMVP candidates is maintained during the encoding / decoding process. When a new CTU row is encountered, the table is reset (cleared). When there is a non-sub-block inter-coded CU, the associated motion information is added to the last item of the table as a new HMVP candidate.

[0106] In VVC, the HMVP table size can be set to 6, that is, a maximum of 6 HMVP candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule can be utilized, where a redundancy check is first applied to find whether the same HMVP exists in the table. If found, the same HMVP is deleted from the table, and then all HMVP candidates can be moved forward.

[0107] In the process of constructing the merge candidate list, the latest HMVP candidates in the table are checked in turn and inserted into the merge candidate list, after the TMVP candidates. Redundancy check can be used to check the HMVP candidates with spatial or temporal merge candidates.

[0108] After inserting the HMVP candidates, if the merge candidate list is not full, pairwise average candidates are added. Pairwise average candidates are generated by averaging the predefined candidate pairs in the existing merge candidate list. The predefined candidate pairs are defined as {(0,1), (0,2), (1,2), (0,3), (1,3), (2,3)}, where the numbers represent the merge index in the merge candidate list. The average motion vector is calculated separately for each reference picture list. If two motion vectors are available in one list, they are averaged even if they point to different reference pictures. If only one motion vector is available, the available motion vector is used directly. If no motion vector is available, the list is considered invalid.

[0109] When the merge list is still not full after adding pairwise average merge candidates, a zero motion vector is inserted at the end until the maximum number of merge candidates is reached.

[0110] In VVC, in addition to the conventional merge mode, the construction of the merge candidate list can also be used for the merge modes of motion vector difference mode (MMVD) and triangle partition mode (TPM).

[0111] In MMVD, the merge candidate phase is first selected from the merge candidate list, and then further refined by the signal motion vector difference (MVD) information. The size of the MMVD merge candidate list is set to 2. The merge candidate flag can be identified to specify which of the two MMVD candidates is used as the reference motion vector (MV). The MVD information can be represented by a distance indicator and a direction indicator. The distance indicator specifies the motion amplitude information and indicates a preset offset from the reference motion vector MV. The relationship between the distance indicator and the preset offset is as follows Figure 6 The direction indicator specifies the sign of the offset added to the reference motion vector MV, for example, 0 indicates a positive sign and 1 indicates a negative sign.

[0112] In TPM, the CU is evenly split into two triangular partitions using diagonal or anti-diagonal partitioning. Each triangular partition in the CU can use its own motion for inter-frame prediction. Only a single prediction is allowed for each partition. That is, each partition has one motion vector and one reference indicator. A single prediction motion constraint is applied to ensure that only two motion compensated predictions are required for each CU, similar to bidirectional prediction. If the current CU uses triangular partitioning mode, a flag can be further used to indicate the direction of the triangular partition (diagonal or anti-diagonal), as well as two merge indices (one for each partition). After predicting each triangular partition, a blending process with adaptive weights is used to adjust the sample values ​​along the diagonal or anti-diagonal edges. This corresponds to the prediction signal for the entire CU, and the transform and quantization process can be applied to the entire CU as in conventional inter mode. A merge candidate list can be constructed. The maximum number of TPM merge candidates can be explicitly indicated in the slice header and set to 5 in the CTC.

[0113] In VVC, a merge candidate list is constructed, including spatial candidates, temporal candidates, HMVP, and pairwise average candidates. Different merge list sizes are applied for different inter-frame modes. For example, spatial merge candidates can be inserted into the merge list in the order of {A1, B1, B0, A0, B2}. However, the construction process of spatial merge candidates has not changed from HEVC to VVC, and it does not take into account the new merge candidates (such as HMVP, pairwise average candidates) and new inter-frame modes (such as MMVD, TPM) in VVC. This leads to various shortcomings of the current spatial merge candidates.

[0114] For example, the order of spatial merging candidates can be improved. The number of spatial merging candidates can be adjusted. In addition, for normal mode, MMVD and TPM mode, the construction of spatial merging candidates is fixed, which limits the potential of the merging method. In addition, for low-latency pictures and non-low-latency pictures, the construction of spatial merging candidates is fixed, reducing flexibility. To address the above and other issues, this application provides various solutions.

[0115] For example, in some embodiments, the order of the spatial merging candidates may be changed. A new spatial merging candidate sequence {B1, A1, B0, A0, B2} may be applied. The positions of the spatially adjacent blocks B1, A1, B0, A0, and B2 are as follows: Figure 7 In some embodiments, the order can be changed to {B1, A1, B0, A0, B2}.

[0116] The new order corresponds to the upper neighboring block, the left neighboring block, the upper neighboring block, the left neighboring block, and the upper left neighboring block, which alternates between the upper neighboring block and the left neighboring block. In addition, the new spatial merging candidate order can achieve higher coding performance. Figure 8 and Figure 9 According to some embodiments, the proposed method can achieve an average coding gain of 0.05% and 0.21% in random access (RA) and low delay (LD) configurations, respectively, compared to VTM-6.

[0117] In some embodiments, the number of spatial merge candidates can be changed. In order to achieve a better trade-off between computational complexity and coding performance, it is proposed to reduce the number of spatial merge candidates and apply it in various embodiments of the present application. When the number of spatial merge candidates is limited to 2, the construction order {B1, A1} can be applied. For example, if the adjacent block B1 is available, it can be selected and inserted into the merge list. Then, if the adjacent block A1 is available and different from B1, it can be selected and inserted into the merge list. After inserting the spatial merge candidate {B1, A1}, the following TMVP, HMVP and pairwise average candidates can be added to the merge list.

[0118] When the number of spatial merging candidates is limited to 3, the construction order {B1, A1, B0} can be applied. The order of checking adjacent blocks is B1->A1->B0, and if the corresponding MV is available and not redundant, it can be inserted into the merge list.

[0119] When using the spatial merging candidate {B1, A1, B0}, the experimental results compared with VTM-6 are shown in Figure 10 and Figure 11 According to some embodiments, the proposed technique can achieve coding gains of 0.00% and 0.10% in RA and LD configurations.

[0120] In addition, in some VVC technologies, the total number of merge candidates can be marked. In some embodiments of the present disclosure, it is recommended that the number of spatial merge candidates be additionally marked to provide greater flexibility in the process of constructing the merge candidate list. Taking into account the prediction structure of the current picture, the number of spatial merge candidates can be set to different values. If the current picture is a non-low-latency picture, the number of spatial merge candidates can be set to a first value. A non-low-latency picture may refer to a picture that is encoded using reference pictures from the past and the future according to the display order. Otherwise, if the current picture is a low-latency picture, the number of spatial merge candidates can be set to a second value. A low-latency picture may refer to a picture that is encoded using only past reference pictures according to the display order. The first value may be greater than the second value. The first value and the second value may be explicitly marked in the bitstream, for example in a slice header. Figure 12 An example is shown in .

[0121] The syntax element num_spatial_merge_cand_minus2 (e.g. Figure 12 Element 1201 in num_spatial_merge_cand_minus2 may indicate the number of spatial merge candidates for the current segment. The value of num_spatial_merge_cand_minus2 may be in the range of 0 to 3, inclusive. When the element num_spatial_merge_cand_minus2 is not present, it may be inferred to be equal to 0.

[0122] Depending on the reference pictures used to encode the current segment, a segment can be classified as low-delay or non-low-delay and a different number of merge candidates can be used. The value of num_spatial_merge_cand_minus2 can be set accordingly by the encoder and signaled in the bitstream.

[0123] In addition, instead of marking one syntax element in the slice header, the two syntax elements num_spatial_merge_cand_minus2_non_lowdelay and num_spatial_merge_cand_minus2_lowdelay can be marked in the picture parameter set (PPS) or sequence parameter set (SPS), such as Figure 13 As shown, Figure 13 (e.g., element 1301) and Figure 14 (For example, element 1401.) Furthermore, at the segment level, depending on the segment type, a corresponding number of spatial merging candidates may be used.

[0124] The values ​​of num_spatial_merge_cand_minus2_non_lowdelay and num_spatial_merge_cand_minus2_lowdelay may represent the number of spatial merge candidates for non-low-delay and low-delay segments, respectively. The values ​​of num_spatial_merge_cand_minus2_non_lowdelay and num_spatial_merge_cand_minus2_lowdelay may be in the range of 0 to 3, inclusive. When num_spatial_merge_cand_minus2_non_lowdelay or num_spatial_merge_cand_minus2_lowdelay is not present, it may be inferred to be equal to 0.

[0125] In some embodiments, separate construction orders of spatial merge candidates can be applied to different inter-frame modes. For example, two construction orders of spatial merge candidates can be considered, including {B1, A1, B0, A0, B2} and {A1, B1, B0, A0, B2}. Different construction orders can be used for normal merge mode, MMVD mode and TPM mode. In some embodiments, it is recommended to use {B1, A1, B0, A0, B2} for normal merge mode and TPM mode, and to use {A1, B1, B0, A0, B2} for MMVD mode. The experimental results of the exemplary embodiment are shown in FIG. Figure 15 and Figure 16 The results show that the average coding gains of this method are 0.07% and 0.16% under RA and LD configurations, respectively.

[0126] Based on this application, those skilled in the art will appreciate that other combinations of spatial merge candidate orders and merge modes may be used. For example, {B1, A1, B0, A0, B2} may be used only for regular merge mode, while {A1, B1, B0, A0, B2} may be used for MMVD mode and TMP mode.

[0127] In some embodiments, an adaptive construction order of spatial merge candidates can be applied based on the frame type. For example, for different types of inter-frame coded pictures, such as low-latency and non-low-latency pictures, different spatial merge candidate construction methods can be applied. In some embodiments, for low-latency pictures, the construction order of spatial merge candidates {B1, A1, B0, A0, B2} can be used for normal merge mode, TPM mode and MMVD mode. For non-low-latency pictures, the construction order of spatial merge candidates {B1, A1, B0, A0, B2} can be used for normal merge mode and TPM mode, and the construction order of spatial merge candidates {A1, B1, B0, A0, B2} can be used for MMVD mode. The experimental results of the exemplary embodiment are shown in FIG. Figure 17 and Figure 18 The results show that the average coding gains of this method are 0.08% and 0.21% under RA and LD configurations, respectively.

[0128] Figure 19 1900 is a flowchart of an exemplary video processing method 1900 consistent with an embodiment of the present disclosure. In some embodiments, the method 1900 may be performed by an encoder, a decoder, a device (e.g., Figure 4A For example, a processor (e.g., Figure 4A The processor 402 of the computer 100 may perform the method 1900. In some embodiments, the method 1900 may be implemented by a computer program product contained in a computer-readable medium, including a computer (e.g., Figure 4A Computer-executable instructions, such as program codes, executed by the device 400).

[0129] In step 802, a set of spatial merge candidates may be inserted into a merge candidate list of a coding block, for example, by an encoder, a decoder, or a device (e.g., a device Figure 4A 400). In VVC, the order of spatial merging candidates can be improved. Spatial merging candidates can be inserted in the following order: upper neighboring block, left neighboring block, upper neighboring block, left neighboring block and upper left neighboring block. For example, Figure 7 The new order of spatial merging candidates {B1, A1, B0, A0, B2} is shown in .

[0130] The number of spatial merging candidates can be adjusted. In step 804, a preset number limit of spatial merging candidates is determined.

[0131] In step 806, if the number is limited to 2, the spatial merge candidates are inserted into the merge candidate list based on the following order: upper neighboring block, left neighboring block. When the number of spatial merge candidates is limited to 2, the construction order {B1, A1} can be applied. For example, if neighboring block B1 is available, it can be selected and inserted into the merge list. Then, if neighboring block A1 is available and different from B1, it can be selected and inserted into the merge list. After inserting the spatial merge candidates {B1, A1}, the following TMVP, HMVP, and pairwise average candidates can be added to the merge list.

[0132] In step 808, if the number is limited to 3, spatial merge candidates are inserted into the merge candidate list in the following order: upper neighboring block, left neighboring block, upper neighboring block. When the number of spatial merge candidates is limited to 3, the construction order {B1, A1, B0} can be applied. Neighboring blocks are checked in the order B1->A1->B0. If available and non-redundant, the corresponding MV is inserted into the merge list.

[0133] In some embodiments, at least one of the following may be added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

[0134] In HMVP, the motion information of previously coded blocks is stored in a FIFO table and used as the motion vector predictor for the current coding unit. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. When a non-subblock inter-coded coding unit exists, the motion information associated with the non-subblock inter coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0135] The pairwise average candidate is generated by averaging candidate pairs in the merge candidate list and is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0136] When the merge list is still not full after adding pairwise average merge candidates, zero motion vectors are inserted to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0137] In step 810, the encoder or decoder determines whether to apply the first coding mode or the second coding mode to the coding block. The first coding mode is different from the second coding mode. In some embodiments, each of the first coding mode and the second coding mode can be one of a normal merge mode, a merge mode with motion vector difference (MMVD), and a triangle partitioning mode (TPM).

[0138] In step 812, when the first coding mode is applied to the coding block, spatial merge candidates are inserted according to the first construction order. For example, in MMVD, a merge candidate is first selected from the merge candidate list, and then refined by marking the motion vector difference (MVD) information, and by marking the merge candidate flag to specify which of the two MMVD candidates is used as the basic motion vector. The MVD information can be represented by a distance index and a direction index. The distance index specifies the motion magnitude information and indicates a predefined offset from the basic MV. The relationship between the distance index and the preset offset is as follows: Figure 6 The direction index specifies the sign of the offset added to the base MV, for example, 0 for positive and 1 for negative.

[0139] In step 814, when the second coding mode is applied to the coding block, spatial merging candidates are inserted according to the second construction order. For example, in TPM, the coding unit is evenly divided into two triangular partitions using at least one of diagonal partitioning or anti-diagonal partitioning. Each triangular partition in the CU can use its own motion for inter-frame prediction. Each partition only allows a single prediction. That is, each partition has a motion vector and a reference index.

[0140] In step 816 , it is determined whether the coding block is part of a low-latency picture or a non-low-latency picture.

[0141] In step 818, when the coding block is part of a low-latency picture, spatial merging candidate candidates are inserted according to the third construction order. In some embodiments, for low-latency pictures, the construction order of spatial merging candidates {B1, A1, B0, A0, B2} can be used for normal merge mode, TPM mode, and MMVD mode.

[0142] In step 820, when the coding block is part of a non-low latency picture, spatial merge candidates are inserted according to the fourth build order. The third build order is different from the fourth build order. The third build order and the fourth build order are used for merge mode with motion vector difference. In some embodiments, for non-low latency pictures, the build order of spatial merge candidates {B1, A1, B0, A0, B2} can be used for normal merge mode and TPM mode, and the build order of spatial merge candidates {A1, B1, B0, A0, B2} can be used for MMVD mode.

[0143] Consistent with the present application, those skilled in the art will appreciate that one or more of the above methods may be used in combination or individually. For example, the technique of using a reduced number of spatial merging candidates may be combined with the proposed method of using spatial merging candidates in different construction orders for different modes.

[0144] In some embodiments, a non-transitory computer-readable storage medium including instructions is also provided, and these instructions can be executed by a device for performing the above method (e.g., an encoder and a decoder disclosed in the present application). Common forms of non-transitory media include, for example, a floppy disk, a floppy disk, a hard disk, a solid-state drive, a tape or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with a hole pattern, RAM, PROM and EPROM, FLASH-EPROM or any other flash memory, NVRAM, a cache, a register, any other memory chip or cartridge memory, and a network version of the same. The device may include one or more processors (CPU), an input / output interface, a network interface, and / or a memory.

[0145] It should be noted that the relational terms such as "first" and "second" in this document are used only to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "including", "having", "comprising" and "including" and other similar forms have the same meaning and are open-ended, as the one or more items following any of these words are not intended to be an exhaustive list of such items or to be limited to the listed items.

[0146] As used herein, unless expressly stated otherwise, the term "or" encompasses all possible combinations unless not feasible. For example, if it is stated that a component can include A or B, then unless expressly stated otherwise or not feasible, the component can include A, or B, or A and B. As a second example, if it is stated that a component can include A, B, or C, then unless expressly stated otherwise or not feasible, the component can include A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0147] It will be appreciated that the above embodiments may be implemented by hardware or software (program code) or a combination of hardware and software. If implemented by software, it may be stored in the above-mentioned computer-readable medium. When executed by a processor, the software may execute the disclosed method. The computing units and other functional units described in this application may be implemented by hardware, by software, or by a combination of hardware and software. Those of ordinary skill in the art may also appreciate that the above-mentioned multiple modules / units may be combined into one module / unit, and each of the above-mentioned modules / units may be further divided into multiple submodules / subunits.

[0148] The above embodiments can be further described using the following terms:

[0149] 1. A video processing method, comprising:

[0150] Insert a set of spatial merge candidates into the merge candidate list of the coding block,

[0151] The spatial merging candidates are inserted in the following order: upper adjacent block, left adjacent block, upper adjacent block, left adjacent block and upper left adjacent block.

[0152] 2. The method according to clause 1, further comprising:

[0153] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

[0154] 3. The method of clause 2, wherein motion information of a previously coded block is stored in the FIFO table and used as a motion vector predictor for the current coding unit.

[0155] 4. A method according to any of clauses 2 and 3, wherein the motion information associated with the non-sub-block inter coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0156] 5. The method according to clause 2, wherein:

[0157] The pairwise average candidates are generated by averaging candidate pairs in the merge candidate list and are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0158] 6. The method according to clause 2, wherein:

[0159] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0160] 7. A video processing method, comprising:

[0161] According to the preset limit on the number of spatial merging candidates, a set of spatial merging candidates is inserted into the merging candidate list of the coding block.

[0162] Wherein, if the number is limited to 2, the set of spatial merging candidates is inserted into the merging candidate list in the following order: upper adjacent block, left adjacent block; and

[0163] If the number is limited to 3, the group of spatial merging candidates is inserted into the merging candidate list in the following order: upper adjacent block, left adjacent block, upper adjacent block.

[0164] 8. The method according to clause 7, wherein

[0165] In response to encoding a current picture using a past reference picture and a future reference picture according to a display order, setting the number of spatial merging candidates to a first value; and

[0166] In response to encoding a current picture using a past reference picture according to a display order, the number of spatial merging candidates is set to a second value smaller than the first value.

[0167] 9. The method according to clause 7, further comprising:

[0168] Mark the number of spatial merging candidates inserted into the merging candidate list.

[0169] 10. The method according to any of clauses 7-9, further comprising:

[0170] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a FIFO table, a pairwise average candidate, or a zero motion vector.

[0171] 11. The method according to clause 10, wherein:

[0172] The motion information of the previously coded block is stored in the FIFO table and used as the motion vector predictor of the current coding unit.

[0173] 12. A method according to any one of clauses 10 and 11, wherein

[0174] The motion information associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0175] 13. The method according to clause 10, wherein:

[0176] Pairwise average candidates are generated by averaging candidate pairs in a merge candidate list, and are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0177] 14. The method according to clause 10, wherein:

[0178] The zero motion vector is inserted at the end of the merge candidate list until a maximum number of merge candidates is reached.

[0179] 15. A video processing method, comprising:

[0180] Insert a set of spatial merge candidates into the merge candidate list of the coded block, where:

[0181] When applying the first coding mode to the coding block, inserting the set of spatial merging candidates according to a first construction order; and

[0182] When applying the second coding mode to the coding block, inserting the set of spatial merging candidates according to the second construction order,

[0183] The first build order is different from the second build order.

[0184] 16. The method of clause 15, wherein the first encoding mode and the second encoding mode are two different modes selected from the following modes:

[0185] Normal merge mode, merge motion vector difference mode (MMVD) and triangle partition mode (TPM).

[0186] 17. The method according to clause 16, wherein

[0187] In MMVD, a merge candidate is first selected from the merge candidate list, then refined by the marked motion vector difference (MVD) information, and the merge candidate flag is marked to specify which of the two MMVD candidates is used as the base motion vector.

[0188] 18. The method according to clause 16, wherein:

[0189] In TPM, a coding unit is evenly partitioned into two triangular partitions using at least one of diagonal partitioning or anti-diagonal partitioning.

[0190] 19. The method according to any one of clauses 15 and 16, further comprising:

[0191] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor from a FIFO table, a pairwise average candidate, or a zero motion vector.

[0192] 20. The method according to clause 19, wherein

[0193] The motion information of the previously coded block is stored in a FIFO table and used as a motion vector predictor for the current coding unit.

[0194] 21. A method according to any one of clauses 19 and 20, wherein

[0195] The motion information associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0196] 22. The method according to clause 19, wherein

[0197] Pairwise average candidates are generated by averaging candidate pairs in the merge candidate list, and are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0198] 23. The method according to clause 19, wherein:

[0199] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0200] 24. A video processing method, comprising:

[0201] Insert a set of spatial merge candidates into the merge candidate list of the coded block, where:

[0202] When the coded block is part of a low-latency picture, inserting the set of spatial merging candidates in a first construction order; and

[0203] When the coded block is part of a non-low-latency picture, the set of spatial merging candidates is inserted according to the second construction order,

[0204] The first build order is different from the second build order.

[0205] 25. The method according to clause 24, wherein:

[0206] The first construction order and the second construction order are used for a motion vector difference (MMVD) merge mode.

[0207] 26. The method according to clause 24, further comprising:

[0208] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a FIFO table, a pairwise average candidate, or a zero motion vector.

[0209] 27. The method according to clause 26, wherein

[0210] The motion information of the previously coded block is stored in a FIFO table and used as a motion vector predictor for the current coding unit.

[0211] 28. A method according to any one of clauses 26 and 27, wherein

[0212] The motion information associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0213] 29. The method according to clause 26, wherein

[0214] The pairwise average candidate is generated by averaging candidate pairs in the merge candidate list and is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0215] 30. The method according to clause 26, wherein:

[0216] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0217] 31. A video processing device, comprising:

[0218] a memory storing a set of instructions; and

[0219] One or more processors configured to execute a set of instructions to cause the apparatus to:

[0220] Insert a set of spatial merge candidates into the merge candidate list of the coding block,

[0221] The set of spatial merging candidates is inserted in the following order: upper neighboring block, left neighboring block, upper neighboring block, left neighboring block and upper-left neighboring block.

[0222] 32. The apparatus of clause 31, wherein the one or more processors are configured to execute the set of instructions to cause the apparatus to further perform:

[0223] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

[0224] 33. The apparatus according to clause 32, wherein

[0225] The motion information of the previously coded block is stored in the FIFO table and used as the motion vector predictor of the current coding unit.

[0226] 34. The apparatus according to any one of clauses 32 and 33, wherein

[0227] The motion information candidate associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP.

[0228] 35. The apparatus of clause 32, wherein:

[0229] The pairwise average candidate is generated by averaging candidate pairs in the merge candidate list, and is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0230] 36. The apparatus of clause 32, wherein:

[0231] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0232] 37. A video processing device comprising:

[0233] a memory storing a set of instructions; and

[0234] one or more processors configured to execute a set of instructions to cause the apparatus to:

[0235] Based on the preset number limit of spatial merging candidates, a set of spatial merging candidates is inserted into the merging candidate list of the coding block,

[0236] Wherein, if the number is limited to 2, the set of spatial merging candidates is inserted into the merging candidate list in the following order: upper adjacent block, left adjacent block; and

[0237] If the number is limited to 3, the set of spatial merging candidates is inserted into the merging candidate list in the following order: upper adjacent block, left adjacent block, upper adjacent block.

[0238] 38. The apparatus of clause 37, wherein

[0239] In response to encoding a current picture using a past reference picture and a future reference picture according to a display order, setting the number of spatial merging candidates to a first value; and

[0240] In response to the current picture having been encoded using a past reference picture according to a display order, the number of spatial merging candidates is set to a second value that is smaller than the first value.

[0241] 39. The apparatus of clause 37, wherein the one or more processors are configured to execute the set of instructions to cause the apparatus to further perform:

[0242] Marks the number of spatial merge candidates inserted in the merge candidate list.

[0243] 40. An apparatus according to any of clauses 37-39, wherein the one or more processors are configured to execute the set of instructions to cause the apparatus to further perform:

[0244] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a FIFO table, a pairwise average candidate, or a zero motion vector.

[0245] 41. The apparatus of clause 40, wherein

[0246] The motion information of the previously coded block is stored in a FIFO table and used as a motion vector predictor for the current coding unit.

[0247] 42. Apparatus according to any one of clauses 40 and 41, wherein

[0248] The motion information associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0249] 43. The apparatus of clause 40, wherein:

[0250] The pairwise average candidate is generated by averaging candidate pairs in the merge candidate list, and is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0251] 44. The apparatus of clause 40, wherein:

[0252] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0253] 45. A video processing device comprising:

[0254] a memory storing a set of instructions; and

[0255] one or more processors configured to execute a set of instructions to cause the apparatus to:

[0256] Insert a set of spatial merge candidates into the merge candidate list of the coded block, where:

[0257] When applying the first coding mode to the coding block, inserting the set of spatial merging candidates according to a first construction order; and

[0258] When applying the second coding mode to the coding block, inserting the set of spatial merging candidates according to the second construction order,

[0259] The first build order is different from the second build order.

[0260] 46. ​​The apparatus of clause 45, wherein the first encoding mode and the second encoding mode are two different modes selected from the following modes:

[0261] Normal merge mode, merge motion vector difference mode (MMVD) and triangle partition mode (TPM).

[0262] 47. The apparatus of clause 46, wherein:

[0263] In MMVD, a merge candidate is first selected from the merge candidate list, then refined by the marked motion vector difference (MVD) information, and the merge candidate flag is marked to specify which of the two MMVD candidates is used as the base motion vector.

[0264] 48. Apparatus according to any one of clauses 45 and 46, wherein

[0265] In TPM, a coding unit is evenly partitioned into two triangular partitions using at least one of diagonal partitioning or anti-diagonal partitioning.

[0266] 49. The apparatus of clause 46, wherein

[0267] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor from a FIFO table, a pairwise average candidate, or a zero motion vector.

[0268] 50. The apparatus of clause 49, wherein:

[0269] The motion information of the previously coded block is stored in a FIFO table and used as a motion vector predictor for the current coding unit.

[0270] 51. Apparatus according to any one of clauses 49 and 50, wherein

[0271] The motion information associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0272] 52. The apparatus of clause 49, wherein:

[0273] Pairwise average candidates are generated by averaging candidate pairs in the merge candidate list, and are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0274] 53. The apparatus of clause 49, wherein:

[0275] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0276] 54. A video processing device comprising:

[0277] a memory storing a set of instructions; and

[0278] one or more processors configured to execute a set of instructions to cause the apparatus to:

[0279] Insert a set of spatial merge candidates into the merge candidate list of the coded block, where:

[0280] When the coded block is part of a low-latency picture, inserting the set of spatial merging candidates in a first construction order; and

[0281] When the coded block is part of a non-low-latency picture, the set of spatial merging candidates is inserted according to the second construction order,

[0282] The first build order is different from the second build order.

[0283] 55. The apparatus of clause 54, wherein:

[0284] The first construction order and the second construction order are used for a motion vector difference (MMVD) merge mode.

[0285] 56. The apparatus of clause 54, wherein the one or more processors are configured to execute the set of instructions to cause the apparatus to further perform:

[0286] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a FIFO table, a pairwise average candidate, or a zero motion vector.

[0287] 57. The apparatus of clause 56, wherein:

[0288] The motion information of the previously coded block is stored in a FIFO table and used as a motion vector predictor for the current coding unit.

[0289] 58. Apparatus according to any one of clauses 56 and 57, wherein

[0290] The motion information associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0291] 59. The apparatus of clause 56, wherein:

[0292] The pairwise average candidate is generated by averaging candidate pairs in the merge candidate list and is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0293] 60. The apparatus of clause 56, wherein:

[0294] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0295] 61. A non-transitory computer-readable medium storing a set of instructions, the set of instructions being executable by at least one processor of a computer to cause the computer to perform a video processing method, the method comprising:

[0296] Insert a set of spatial merge candidates into the merge candidate list of the coding block,

[0297] The set of spatial merging candidates is inserted in the following order: upper neighboring block, left neighboring block, upper neighboring block, left neighboring block and upper-left neighboring block.

[0298] 62. The non-transitory computer-readable medium of clause 61, wherein the computer-executable set of instructions causes the computer to further perform:

[0299] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

[0300] 63. The non-transitory computer-readable medium of clause 62, wherein:

[0301] The motion information of the previously coded block is stored in the FIFO table and used as the motion vector predictor of the current coding unit.

[0302] 64. The non-transitory computer-readable medium of any one of clauses 62 and 63, wherein:

[0303] The motion information candidate associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP.

[0304] 65. The non-transitory computer-readable medium of clause 62, wherein:

[0305] The pairwise average candidate is generated by averaging candidate pairs in the merge candidate list, and is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0306] 66. The non-transitory computer-readable medium of clause 62, wherein:

[0307] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0308] 67. A non-transitory computer-readable medium storing a set of instructions, the set of instructions being executable by at least one processor of a computer to cause the computer to perform a video processing method, the method comprising:

[0309] Based on the preset number limit of spatial merging candidates, a set of spatial merging candidates is inserted into the merging candidate list of the coding block,

[0310] Wherein, if the number is limited to 2, the set of spatial merging candidates is inserted into the merging candidate list in the following order: upper adjacent block, left adjacent block; and

[0311] If the number is limited to 3, the set of spatial merging candidates is inserted into the merging candidate list in the following order: upper adjacent block, left adjacent block, upper adjacent block.

[0312] 68. The non-transitory computer-readable medium of clause 67, wherein:

[0313] In response to encoding a current picture using a past reference picture and a future reference picture according to a display order, setting the number of spatial merging candidates to a first value; and

[0314] In response to the current picture having been encoded using a past reference picture according to a display order, the number of spatial merging candidates is set to a second value that is smaller than the first value.

[0315] 69. The non-transitory computer-readable medium of clause 67, wherein the computer-executable set of instructions causes the computer to further perform:

[0316] Marks the number of spatial merge candidates inserted in the merge candidate list.

[0317] 70. The non-transitory computer-readable medium of any of clauses 67-69, wherein at least one processor is configured to execute the set of instructions to cause the computer to further perform:

[0318] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a FIFO table, a pairwise average candidate, or a zero motion vector.

[0319] 71. The non-transitory computer-readable medium of clause 70, wherein:

[0320] The motion information of the previously coded block is stored in a FIFO table and used as a motion vector predictor for the current coding unit.

[0321] 72. The non-transitory computer-readable medium of any one of clauses 70 and 71, wherein

[0322] The motion information associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0323] 73. The non-transitory computer-readable medium of clause 70, wherein:

[0324] The pairwise average candidate is generated by averaging candidate pairs in the merge candidate list, and is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0325] 74. The non-transitory computer-readable medium of clause 70, wherein:

[0326] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0327] 75. A non-transitory computer-readable medium storing a set of instructions, the set of instructions being executable by at least one processor of a computer to cause the computer to perform a video processing method, the method comprising:

[0328] Insert a set of spatial merge candidates into the merge candidate list of the coded block, where:

[0329] When applying the first coding mode to the coding block, inserting the set of spatial merging candidates according to a first construction order; and

[0330] When applying the second coding mode to the coding block, inserting the set of spatial merging candidates according to the second construction order,

[0331] The first build order is different from the second build order.

[0332] 76. The non-transitory computer-readable medium of clause 75, wherein the first encoding mode and the second encoding mode are two different modes selected from the following modes:

[0333] Normal merge mode, merge motion vector difference mode (MMVD) and triangle partition mode (TPM).

[0334] 77. The non-transitory computer-readable medium of clause 76, wherein:

[0335] In MMVD, a merge candidate is first selected from the merge candidate list, then refined by the marked motion vector difference (MVD) information, and the merge candidate flag is marked to specify which of the two MMVD candidates is used as the base motion vector.

[0336] 78. The non-transitory computer-readable medium of clause 76, wherein:

[0337] In TPM, a coding unit is evenly partitioned into two triangular partitions using at least one of diagonal partitioning or anti-diagonal partitioning.

[0338] 79. The non-transitory computer-readable medium of any one of clauses 75 and 76, wherein

[0339] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor from a FIFO table, a pairwise average candidate, or a zero motion vector.

[0340] 80. The non-transitory computer-readable medium of clause 79, wherein:

[0341] The motion information of the previously coded block is stored in a FIFO table and used as a motion vector predictor for the current coding unit.

[0342] 81. The non-transitory computer-readable medium of any one of clauses 79 and 80, wherein

[0343] The motion information associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0344] 82. The non-transitory computer-readable medium of 79, wherein:

[0345] Pairwise average candidates are generated by averaging candidate pairs in the merge candidate list, and are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0346] 83. The non-transitory computer-readable medium of clause 79, wherein:

[0347] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0348] 84. A non-transitory computer-readable medium storing a set of instructions, the set of instructions being executable by at least one processor of a computer to cause the computer to perform a video processing method, the method comprising:

[0349] Insert a set of spatial merge candidates into the merge candidate list of the coded block, where:

[0350] When the coded block is part of a low-latency picture, inserting the set of spatial merging candidates in a first construction order; and

[0351] When the coded block is part of a non-low-latency picture, the set of spatial merging candidates is inserted according to the second construction order,

[0352] The first build order is different from the second build order.

[0353] 85. The non-transitory computer-readable medium of clause 84, wherein:

[0354] The first construction order and the second construction order are used for a motion vector difference (MMVD) merge mode.

[0355] 86. The non-transitory computer-readable medium of clause 84, wherein the computer is executable by a set of instructions that cause the computer to further perform:

[0356] At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a FIFO table, a pairwise average candidate, or a zero motion vector.

[0357] 87. The non-transitory computer-readable medium of clause 86, wherein:

[0358] The motion information of the previously coded block is stored in a FIFO table and used as a motion vector predictor for the current coding unit.

[0359] 88. The non-transitory computer-readable medium of any one of clauses 86 and 87, wherein:

[0360] The motion information associated with the non-sub-block inter-coded coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0361] 89. The non-transitory computer-readable medium of clause 86, wherein:

[0362] The pairwise average candidate is generated by averaging candidate pairs in the merge candidate list and is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

[0363] 90. The non-transitory computer-readable medium of clause 86, wherein

[0364] Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

[0365] In the foregoing description, embodiments have been described with reference to many specific details, which may vary depending on the implementation. Certain adjustments and modifications may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art, taking into account the specifications and practices of the present application disclosed herein. The foregoing description and embodiments are intended to be examples only, with the true scope and spirit of the present application being indicated by the claims. The order of steps shown in the figures is also intended to be for illustrative purposes only and is not intended to be limited to any particular order of steps. Therefore, it will be understood by those skilled in the art that these steps may be performed in different orders while implementing the same method.

[0366] In the drawings and the specification, exemplary embodiments have been disclosed. However, many variations and modifications may be made to these embodiments. Therefore, although specific terms are used, they are used in a general and descriptive sense only and not for limitation purposes.

Claims

1. A video processing method, applied to an encoder, comprising: Insert a set of spatial merge candidates into the merge candidate list of the coding block, The spatial merging candidates are inserted in the following order: upper adjacent block, left adjacent block, upper right adjacent block, lower left adjacent block and upper left adjacent block; Among them, the order is: upper adjacent block, left adjacent block, upper right adjacent block, lower left adjacent block and upper left adjacent block are upper adjacent block B1, left adjacent block A1, upper right adjacent block B0, lower left adjacent block A0 and upper left adjacent block B2.

2. The method according to claim 1, further comprising: At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

3. The method according to claim 2, wherein: The motion information of the previously coded block is stored in the FIFO table and used as a motion vector predictor for the current coding unit.

4. The method according to claim 3, wherein: The motion information associated with the non-sub-block inter coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

5. The method according to claim 2, wherein: The pairwise average candidates are generated by averaging candidate pairs in the merge candidate list and are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

6. The method according to claim 2, wherein: Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

7. A video processing device, applied to a decoder, comprising: a memory storing an instruction set; and one or more processors configured to execute the set of instructions to cause the device to: Insert a set of spatial merge candidates into the merge candidate list of the coding block, The spatial merging candidates are inserted in the following order: upper adjacent block, left adjacent block, upper right adjacent block, lower left adjacent block and upper left adjacent block; Among them, the order is: upper adjacent block, left adjacent block, upper right adjacent block, lower left adjacent block and upper left adjacent block are upper adjacent block B1, left adjacent block A1, upper right adjacent block B0, lower left adjacent block A0 and upper left adjacent block B2.

8. The apparatus according to claim 7, wherein The one or more processors are configured to execute the set of instructions to cause the device to further perform: At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

9. The apparatus according to claim 8, wherein The motion information of the previously coded block is stored in the FIFO table and used as a motion vector predictor for the current coding unit.

10. The apparatus according to claim 9, wherein The motion information associated with the non-sub-block inter coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

11. The apparatus according to claim 8, wherein The pairwise average candidates are generated by averaging candidate pairs in the merge candidate list and are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

12. The apparatus according to claim 8, wherein Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

13. A non-transitory computer-readable medium storing a bitstream that can be generated by executing a set of instructions by at least one processor of a computer, wherein execution of the set of instructions causes the processor to perform the following video processing method: Insert a set of spatial merge candidates into the merge candidate list of the coding block, in, The spatial merging candidates are inserted in the following order: upper neighboring block, left neighboring block, upper right neighboring block, lower left neighboring block and upper left neighboring block; Among them, the order is: upper adjacent block, left adjacent block, upper right adjacent block, lower left adjacent block and upper left adjacent block are upper adjacent block B1, left adjacent block A1, upper right adjacent block B0, lower left adjacent block A0 and upper left adjacent block B2.

14. The non-transitory computer-readable medium of claim 13, wherein: The computer-executable instruction set causes the computer to further perform: At least one of the following is added to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

15. The non-transitory computer-readable medium of claim 14, wherein: The motion information of the previously coded block is stored in the FIFO table and used as a motion vector predictor for the current coding unit.

16. The non-transitory computer-readable medium of claim 15, wherein: The motion information associated with the non-sub-block inter coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

17. The non-transitory computer-readable medium of claim 14, wherein: The pairwise average candidates are generated by averaging candidate pairs in the merge candidate list and are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being not full.

18. The non-transitory computer-readable medium of claim 14, wherein: Insert zero motion vectors to the end of the merge candidate list until the maximum number of merge candidates is reached.

Images