Method of constructing merge candidate list

Abstract

Systems and methods are provided for constructing a merge candidate list for video processing. An example method includes inserting a set of spatial merge candidates into a merge candidate list for a coding block, where the set of spatial merge candidates are inserted in the following order: an above neighboring block, a left neighboring block, the above neighboring block, the left neighboring block, and a top-left neighboring block. The method can also include adding at least one of the following to the merge candidate list: a temporal merge candidate from a collocated coding unit, a history-based motion vector predictor (HMVP) from a first-in-first-out (FIFO) table, a pair-wise average candidate, or a zero motion vector.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 902,790, filed September 19, 2019, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Video is a set of still images (or "frames") that capture visual information. To reduce storage memory and transmission bandwidth, video can be compressed before storage or transmission and decompressed before display. The compression process is usually called encoding, and the decompression process is usually called decoding. Currently, there are various video coding formats that use standardized video coding technologies, the most common being those based on prediction, transform, quantization, entropy coding, and in-loop filtering. Video coding standards that specify specific video coding formats, such as the High Efficiency Video Coding (HEVC / H.265) standard, the Universal Video Coding (VVC / H.266) standard, and the AVS standard, are developed by standardization organizations. As more and more advanced video coding technologies are adopted in video standards, the coding efficiency of new video coding standards is also increasing. Summary of the Invention

[0004] This application provides a method for constructing a merge candidate list. According to some embodiments, an exemplary method includes: inserting a set of spatial merge candidates into a merge candidate list of coded blocks, wherein the spatial merge candidates are inserted in the following order: top adjacent block, left adjacent block, top adjacent block, left adjacent block, and top-left adjacent block.

[0005] According to some embodiments, an exemplary method includes: inserting a set of spatial merging candidates into a merging candidate list of coded blocks based on a preset quantity limit. If the quantity is limited to 2, the spatial merging candidates are inserted into the merging candidate list in the following order: top adjacent block, left adjacent block. If the quantity is limited to 3, the spatial merging candidates are inserted into the merging candidate list in the following order: top adjacent block, left adjacent block, top adjacent block.

[0006] According to some embodiments, an exemplary method includes: inserting a set of spatial merging candidates into a merging candidate list of a coding block, wherein: when a first coding mode is applied to the coding block, the spatial merging candidates are inserted in a first construction order; and when a second coding mode is applied to the coding block, the spatial merging candidates are inserted in a second construction order, wherein the first construction order is different from the second construction order.

[0007] According to some embodiments, an exemplary method includes: inserting a set of spatial merging candidates into a merging candidate list of coded blocks, wherein: when the coded block is a low-latency image, the spatial merging candidates are inserted according to a first construction order; when the coded block is a non-low-latency image, the spatial merging candidates are inserted according to a second construction order, the first construction order being different from the second construction order. Attached Figure Description

[0008] The following detailed description and accompanying drawings illustrate embodiments and various aspects of this application. The various features shown in the figures are not drawn to scale.

[0009] Figure 1 The structure of an example video sequence is shown according to some embodiments of this application.

[0010] Figure 2A A schematic diagram illustrating an exemplary encoding process of a hybrid video encoding system consistent with embodiments of this application is shown.

[0011] Figure 2B A schematic diagram of another exemplary encoding process of a hybrid video encoding system consistent with the embodiments of this application is shown.

[0012] Figure 3A A schematic diagram illustrating an exemplary decoding process of a hybrid video coding system consistent with embodiments of this application is shown.

[0013] Figure 3B A schematic diagram of another exemplary decoding process of a hybrid video coding system consistent with the embodiments of this application is shown.

[0014] Figure 4A A block diagram of an exemplary apparatus for encoding or decoding video, consistent with embodiments of this application, is shown.

[0015] Figure 4B Exemplary locations of spatial merging candidates consistent with embodiments of this application are shown.

[0016] Figure 4C An exemplary location of the time merging candidate is shown, consistent with the embodiments of this application.

[0017] Figure 5 An exemplary scaling of time merging candidates consistent with embodiments of this application is shown.

[0018] Figure 6 An exemplary relationship between the distance index and the predefined offset and the motion vector difference (MMVD) is shown in a merging mode consistent with embodiments of this application.

[0019] Figure 7Exemplary locations of spatial merging candidates consistent with embodiments of this application are shown.

[0020] Figure 8 Exemplary experimental results are shown that are consistent with embodiments of this application and compared to VTM-6 under a random access (RA) configuration.

[0021] Figure 9 Exemplary experimental results are shown that are consistent with embodiments of this application and compared to VTM-6 in a low latency (LD) configuration.

[0022] Figure 10 Exemplary experimental results are shown that are consistent with embodiments of this application and compared to VTM-6 under RA configuration.

[0023] Figure 11 Exemplary experimental results are shown that are consistent with embodiments of this application and compared to VTM-6 under LD configuration.

[0024] Figure 12 An exemplary syntax structure for a fragment header consistent with embodiments of this application is shown.

[0025] Figure 13 An exemplary syntax structure for a sequence parameter set (SPS) consistent with embodiments of this application is shown.

[0026] Figure 14 An exemplary syntax structure for a Picture Parameter Set (PPS) consistent with embodiments of this application is shown.

[0027] Figure 15 Exemplary experimental results are shown that are consistent with embodiments of this application and compared to VTM-6 under RA configuration.

[0028] Figure 16 Exemplary experimental results are shown that are consistent with embodiments of this application and compared to VTM-6 under LD configuration.

[0029] Figure 17 Exemplary experimental results are shown that are consistent with embodiments of this application and compared to VTM-6 under RA configuration.

[0030] Figure 18 Exemplary experimental results are shown that are consistent with embodiments of this application and compared to VTM-6 under LD configuration.

[0031] Figure 19 A flowchart of an exemplary video processing method consistent with embodiments of this application is shown. Detailed Implementation

[0032] Reference can now be made to exemplary embodiments, examples of which are shown in the accompanying drawings. The following description refers to the accompanying drawings, wherein the same numbers in different drawings denote the same or similar elements unless otherwise stated. The embodiments described in the following example embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with the aspects related to this application as described in the appended claims. Specific aspects of this application are described in more detail below. In case of conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall prevail.

[0033] Video is composed of frames arranged in a time sequence to store visual information. Video capture devices (e.g., cameras) can be used to capture and store these images in chronological order, and video playback devices (e.g., televisions, computers, smartphones, tablets, video players, or any end-user terminal with a display capability) can be used to display such images in chronological order. Furthermore, in some applications, video capture devices can transmit captured video in real time to video playback devices (e.g., computers with monitors), such as for surveillance, conferencing, or live streaming.

[0034] To reduce the storage space and transmission bandwidth required for such applications, video can be compressed before storage and transmission, and decompressed before display. Compression and decompression can be implemented by software executed by a processor (e.g., a processor in a general-purpose computer) or dedicated hardware. The module used for compression is typically called an "encoder," and the module used for decompression is typically called a "decoder." Encoders and decoders can be collectively referred to as a "codec." Encoders and decoders can be implemented as any of a variety of suitable hardware, software, or combinations thereof. For example, hardware implementations of encoders and decoders can include circuits such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, or any combination thereof. Software implementations of encoders and decoders can include program code, computer-executable instructions, firmware, or any suitable computer-implemented algorithm or process embedded in a computer-readable medium. Video compression and decompression can be implemented using various algorithms or standards, such as MPEG-1, MPEG-2, MPEG-4, H.26x series, etc. In some applications, a codec can decompress video from a first encoding standard and recompress the decompressed video using a second encoding standard; in this case, the codec can be called a "transcoder."

[0035] Video encoding processes identify and retain useful information that can be used to reconstruct images, while ignoring unimportant information during the reconstruction process. If the ignored, unimportant information cannot be fully reconstructed, this encoding process can be called "lossy." Otherwise, it can be called "lossless." Most encoding processes are lossy, a trade-off made to reduce required storage space and transmission bandwidth.

[0036] Useful information about the image being encoded (referred to as the "current image") includes changes relative to a reference image (e.g., a previously encoded and reconstructed image). These changes can include variations in pixel position, brightness, or color, with positional changes being of most interest. The positional changes of a set of pixels representing an object can reflect the object's movement between the reference and current images.

[0037] An image encoded without referencing another image (i.e., it is its own reference image) is called an "I-image". An image encoded using a previous image as a reference image is called a "P-image". An image encoded using both a previous image and a future image as reference images (i.e., the reference is "bidirectional") is called a "B-image".

[0038] To achieve the same subjective quality as HEVC / H.265 using half the bandwidth, JVET has been developing techniques beyond HEVC using the Joint Exploratory Model (JEM) reference software. With the incorporation of coding techniques into JEM, JEM achieves higher coding performance than HEVC.

[0039] The VVC standard further incorporates more coding techniques, providing better compression performance. VVC is based on the same hybrid video coding system used in modern video compression standards such as HEVC, H.264 / AVC, MPEG2, H.263, etc. In VVC, a merge candidate list containing new merge candidates can be constructed. Different sized merge lists are applied for different inter-frame modes. Embodiments in this application consider new merge candidates (e.g., HMVP, Pairwise Average) and new inter-frame modes (e.g., MMVD, TPM) in VVC. For example, the order of spatial merge candidates can be improved, and the number of spatial merge candidates can be adjusted. Furthermore, the structure of the spatial merge candidates is fixed for regular mode, MMVD, and TPM modes.

[0040] Figure 1The illustration shows the structure of an example video sequence 100 according to some embodiments of this application. The video sequence 100 can be live video or video that has been captured and archived. The video 100 can be real-life video, computer-generated video (e.g., computer game video), or a combination thereof (e.g., real-life video with augmented reality effects). The video sequence 100 can be input from a video capture device (e.g., a camera), a video archive containing previously captured video (e.g., a video file stored on a storage device), or a video input interface (e.g., a video broadcast transceiver) to receive video from a video content provider.

[0041] like Figure 1 As shown, video sequence 100 may include a series of images arranged chronologically along a timeline, including images 102, 104, 106, and 108. Images 102-106 are consecutive, with more images between images 106 and 108. Figure 1 In this diagram, image 102 is an I-image, and its reference image is image 102 itself. Image 104 is a P-image, as indicated by the arrow, and its reference image is image 102. Image 106 is a B-image, as indicated by the arrow, and its reference images are images 104 and 108. In some embodiments, the reference image of an image (e.g., image 104) may not be directly located before or after the image. For example, the reference image of image 104 may be an image preceding image 102. It should be noted that the reference images of images 102-106 are merely examples, and this application does not limit the embodiments of the reference images to... Figure 1 The example shown.

[0042] Due to the computational complexity of such tasks, video codecs typically do not encode or decode the entire image at once. Instead, they can segment the image into basic segments and encode or decode each segment sequentially. In this application, these basic segments are referred to as Basic Processing Units (“BPUs”). For example, Figure 1Structure 110 illustrates an example structure of a picture (e.g., any one of pictures 102-108) of video sequence 100. In structure 110, the picture is divided into 4×4 basic processing units, the boundaries of which are shown as dashed lines. In some embodiments, the basic processing unit may be referred to as a “macroblock” in some video coding standards (e.g., MPEG series, H.261, H.263, or H.264 / AVC), or as a “coding tree unit” (“CTU”) in some other video coding standards (e.g., H.265 / HEVC or H.266 / VVC). The basic processing units in the picture can have different sizes, such as 128×128, 64×64, 32×32, 16×16, 4×8, 16×32, or pixels of any shape and size. The size and shape of the basic processing units for the picture can be selected based on a balance between coding efficiency and the level of detail to be retained in the basic processing units.

[0043] A basic processing unit can be a logical unit that may include a set of different types of video data stored in computer memory (e.g., in a video frame buffer). For example, a basic processing unit for a color picture may include a luminance component (Y) representing achromatic luminance information, one or more chrominance components (e.g., Cb and Cr) representing color information, and associated syntax elements, where the luminance and chrominance components may have basic processing units of the same size. In some video coding standards (e.g., H.265 / HEVC or H.266 / VVC), the luminance and chrominance components may be referred to as “code tree blocks” (“CTBs”). Any operation performed on a basic processing unit can be repeated for each of its luminance and chrominance components.

[0044] Video encoding involves multiple operational stages, examples of which are shown in... Figure 2A-2B and Figures 3A-3BAs shown in the diagram. For each stage, the size of the basic processing unit may still be too large to process, and therefore can be further divided into segments referred to herein as "basic processing subunits". In some embodiments, the basic processing subunit may be referred to as a "block" in some video coding standards (e.g., MPEG series, H.261, H.263, or H.264 / AVC), or as a "coding unit" ("CU") in some other video coding standards (e.g., H.265 / HEVC or H.266 / VVC). The basic processing subunit may have the same or smaller size as the basic processing unit. Similar to the basic processing unit, the basic processing subunit is also a logical unit, which may include storage in computer memory (e.g., in a video frame buffer). Any operation performed on the basic processing subunit may be repeated for each of its luma and chroma components. It should be noted that this division can be implemented at further levels depending on processing requirements. It should also be noted that different stages may use different schemes to divide the basic processing units.

[0045] For example, in the pattern determination phase (examples of which are in...) Figure 2B As shown in the diagram, the encoder can decide which prediction mode to use for a basic processing unit (e.g., intra-image prediction or inter-image prediction), even if the basic processing unit is too large to make such a decision. The encoder can break down the basic processing unit into multiple basic processing subunits (e.g., CUs in H.265 / HEVC or H.266 / VVC) and determine the prediction type for each individual basic processing subunit.

[0046] For another example, in the prediction phase (the example is in...) Figure 2A-2B As shown in the diagram, the encoder can perform prediction operations at the level of a basic processing subunit (e.g., a CU). However, in some cases, the basic processing subunit may still be too large to process. The encoder can further break down the basic processing subunit into smaller segments (e.g., referred to as "prediction blocks" or "PBs" in H.265 / HEVC or H.266 / VVC), at which prediction operations can be performed.

[0047] For another example, in the transformation phase (the example of which is in...) Figure 2A-2BAs shown in the diagram, the encoder can perform transform operations on residual basic processing subunits (e.g., CUs). However, in some cases, these basic processing subunits may still be too large to process. The encoder can further divide the basic processing subunits into smaller segments (e.g., referred to as "transform blocks" or "TBs" in H.265 / HEVC or H.266 / VVC), at which transform operations can be performed. It should be noted that the partitioning scheme of the same basic processing subunit can differ between the prediction and transform phases. For example, in H.265 / HEVC or H.266 / VVC, the prediction blocks and transform blocks of the same CU can have different sizes and numbers.

[0048] exist Figure 1 In structure 110, the basic processing unit 112 is further divided into 3×3 basic processing sub-units, the boundaries of which are shown by dashed lines. In different schemes, different basic processing units of the same image can be divided into different basic processing sub-units.

[0049] In some implementations, to provide parallel processing and fault tolerance for video encoding and decoding, an image can be divided into multiple processing regions. This allows the encoding or decoding process for a particular region of the image to be independent of information from any other region. In other words, each region of the image can be processed independently. In this way, the codec can process different regions of the image in parallel, thereby improving encoding efficiency. Furthermore, when data in one region is corrupted during processing or lost during network transmission, the codec can correctly encode or decode other regions of the same image without relying on the corrupted or lost data, thus providing fault tolerance. In some video coding standards, images can be divided into different types of regions. For example, H.265 / HEVC and H.266 / VVC provide two types of regions: “segments” and “pastures.” It should also be noted that different images in the video sequence 100 can have different partitioning schemes for dividing the images into regions.

[0050] For example, in Figure 1 In the diagram, structure 110 is divided into three regions 114, 116, and 118, whose boundaries are shown as solid lines within structure 110. Region 114 comprises four basic processing units. Each of regions 116 and 118 comprises six basic processing units. It should be noted that... Figure 1 The basic processing unit, basic processing sub-unit, and region of structure 110 in this application are merely examples, and this application does not limit its implementation.

[0051] Figure 2A A schematic diagram of an example encoding process 200A consistent with embodiments of this application is shown. For example, encoding process 200A can be performed by an encoder. Figure 2A As shown, the encoder can encode the video sequence 202 into a video stream 228 according to the encoding process 200A. Similar to... Figure 1 Video sequence 100 and video sequence 202 may include a set of images arranged in chronological order (referred to as "original images"). Similar to... Figure 1 In structure 110, the encoder can divide each raw image of video sequence 202 into a basic processing unit, a basic processing subunit, or a region for processing. In some embodiments, the encoder can perform encoding process 200A at the basic processing unit level for each raw image of video sequence 202. For example, the encoder can perform encoding process 200A iteratively, wherein the encoder can encode a basic processing unit in one iteration of process 200A. In some embodiments, the encoder can perform process 200A in parallel for a region (e.g., region 114-118) of each raw image of video sequence 202.

[0052] exist Figure 2A In this process, the encoder can input the basic processing unit (referred to as the "raw BPU") of the original image of video sequence 202 into prediction stage 204 to generate prediction data 206 and prediction BPU 208. The encoder can subtract prediction BPU 208 from the raw BPU to generate residual BPU 210. The encoder can input residual BPU 210 into transform stage 212 and quantization stage 214 to generate quantization transform coefficients 216. The encoder can input prediction data 206 and quantization transform coefficients 216 into binary coding stage 226 to generate video bitstream 228. Components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 can be referred to as the "forward path". During process 200A, after quantization stage 214, the encoder can input quantization transform coefficients 216 into inverse quantization stage 218 and inverse transform stage 220 to generate reconstructed residual BPU 222. The encoder can add the reconstruction residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224, which is used in the next iteration of process 200A in the prediction phase 204. Components 218, 220, 222, and 224 of process 200A can be referred to as the "reconstruction path". The reconstruction path can be used to ensure that both the encoder and decoder use the same reference data for prediction.

[0053] The encoder can iteratively execute process 200A to encode each raw BPU (in the forward path) of the original image and generate a prediction reference 224 for encoding the next raw BPU (in the reconstruction path) of the original image. After encoding all raw BPUs of the original image, the encoder can continue to encode the next image in the video sequence 202.

[0054] Referring to process 200A, the encoder may receive a video sequence 202 generated by a video capture device (e.g., a camera). The term "receive" as used herein may refer to any action in any manner of receiving, inputting, acquiring, retrieving, obtaining, reading, accessing, or using data for input.

[0055] In prediction stage 204, during the current iteration, the encoder can receive the original BPU and prediction reference 224, and perform prediction operations to generate prediction data 206 and prediction BPU 208. Prediction reference 224 can be generated from the reconstruction path of the previous iteration of process 200A. The purpose of prediction stage 204 is to reduce information redundancy by extracting prediction data 206 from prediction data 206 and prediction reference 224 that can be used to reconstruct the original BPU into prediction BPU 208.

[0056] Ideally, the predicted BPU 208 should be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 208 is typically slightly different from the original BPU. To record this difference, after generating the predicted BPU 208, the encoder can subtract it from the original BPU to generate the residual BPU 210. For example, the encoder can subtract the value of the corresponding pixel in the original BPU from the value (e.g., grayscale or RGB value) of the pixel corresponding to the predicted BPU 208. Each pixel in the residual BPU 210 can have a residual value as the result of this subtraction between the corresponding pixel in the original BPU and the predicted BPU 208. Compared to the original BPU, the predicted data 206 and the residual BPU 210 can have fewer bits, but they can be used to reconstruct the original BPU without significantly reducing quality, thus compressing the original BPU.

[0057] To further compress the residual BPU 210, in the transform phase 212, the encoder can reduce its spatial redundancy by decomposing the residual BPU 210 into a set of two-dimensional “fundamental patterns,” each of which is associated with “transform coefficients.” The fundamental patterns can have the same size (e.g., the size of the residual BPU 210). Each fundamental pattern can represent a frequency component of the residual BPU 210 (e.g., the frequency of brightness variation). No fundamental pattern can be reproduced from any combination of any other fundamental patterns (e.g., a linear combination). In other words, the decomposition decomposes the variations of the residual BPU 210 into the frequency domain. This decomposition is analogous to the discrete Fourier transform of a function, where the fundamental patterns are analogous to the basis functions of the discrete Fourier transform (e.g., trigonometric functions), and the transform coefficients are analogous to the coefficients associated with the basis functions.

[0058] Different transform algorithms can use different base modes. Various transform algorithms, such as discrete cosine transform, discrete sine transform, etc., can be used in transform stage 212. The transform in transform stage 212 is reversible. That is, the encoder can recover the residual BPU 210 through the inverse operation of the transform (called the "inverse transform"). For example, to recover the pixels of the residual BPU 210, the inverse transform can be to multiply the value of the corresponding pixel in the base mode by the corresponding correlation coefficient and sum the products to produce a weighted sum. For video coding standards, both the encoder and decoder can use the same transform algorithm (and therefore the same base mode). Therefore, the encoder can only record the transform coefficients, and the decoder can reconstruct the residual BPU 210 from the transform coefficients without receiving the base mode from the encoder. Compared to the residual BPU 210, the transform coefficients can have fewer bits, but they can be used to reconstruct the residual BPU 210 without significantly degrading the quality. Therefore, the residual BPU 210 is further compressed.

[0059] The encoder can further compress the transform coefficients in the quantization stage 214. During the transform process, different fundamental modes can represent different change frequencies (e.g., brightness change frequencies). Since the human eye is generally better at recognizing low-frequency changes, the encoder can ignore information about high-frequency changes without causing a significant degrade in decoding quality. For example, in the quantization stage 214, the encoder can generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (called a "quantization parameter") and rounding the quotient to its nearest integer. This operation converts some transform coefficients of high-frequency fundamental modes to zero, while transform coefficients of low-frequency fundamental modes can be converted to smaller integers. The encoder can ignore zero-valued quantized transform coefficients 216, further compressing the transform coefficients. The quantization process is also reversible, where the quantized transform coefficients 216 can be reconstructed into transform coefficients in the inverse operation of quantization (called "inverse quantization").

[0060] Because the encoder ignores the remainder of such division during rounding operations, quantization stage 214 can be lossy. Typically, quantization stage 214 can cause the greatest information loss in process 200A. The greater the information loss, the fewer bits the quantization transform coefficients 216 may require. To obtain different levels of information loss, the encoder can use different quantization parameter values ​​or any other parameters in the quantization process.

[0061] In the binary encoding stage 226, the encoder may encode the prediction data 206 and the quantization transform coefficients 216 using binary encoding techniques such as entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless or lossy compression algorithm. In some embodiments, in addition to the prediction data 206 and the quantization transform coefficients 216, the encoder may encode other information in the binary encoding stage 226, such as the prediction mode used in the prediction stage 204, the parameters of the prediction operation, the transform type of the transform stage 212, the parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), etc. The encoder may use the output data of the binary encoding stage 226 to generate a video stream 228. In some embodiments, the video stream 228 may be further packaged for network transmission.

[0062] Referring to the reconstruction path of process 200A, in the inverse quantization stage 218, the encoder can perform inverse quantization on the quantized transform coefficients 216 to generate reconstructed transform coefficients. In the inverse transform stage 220, the encoder can generate reconstructed residual BPU 222 based on the reconstructed transform coefficients. The encoder can add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a prediction reference 224 that will be used in the next iteration of process 200A.

[0063] It should be noted that other variations of process 200A can also be used to encode video sequence 202. In some embodiments, the encoder may execute the various stages of process 200A in a different order. In some embodiments, one or more stages of process 200A may be combined into a single stage. In some embodiments, a single stage of process 200A may be divided into multiple stages. For example, transform stage 212 and quantization stage 214 may be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, one or more stages in FIG. 2 may be omitted from process 200A.

[0064] Figure 2B A schematic diagram of another example encoding process 200B consistent with embodiments of this application is shown. Process 200B can be modified from process 200A. For example, process 200B can be used by an encoder conforming to a hybrid video coding standard (e.g., H.26x series). Compared to process 200A, the forward path of process 200B additionally includes a mode determination stage 230 and divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B additionally includes a loop filter stage 232 and a buffer 234.

[0065] Generally, prediction techniques can be categorized into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-image prediction or "intra-frame prediction") uses pixels from one or more encoded neighboring BPUs within the same image to predict the current BPU. That is, the prediction reference 224 in spatial prediction can include neighboring BPUs. Spatial prediction can reduce the inherent spatial redundancy of images. Temporal prediction (e.g., inter-image prediction or "inter-frame prediction") uses regions from one or more encoded images to predict the current BPU. That is, the prediction reference 224 in temporal prediction can include encoded images. Temporal prediction can reduce the inherent temporal redundancy of images.

[0066] In reference process 200B, during the forward path, the encoder performs prediction operations in spatial prediction phase 2042 and temporal prediction phase 2044. For example, in spatial prediction phase 2042, the encoder may perform intra-frame prediction. For the original BPU of the picture being encoded, prediction reference 224 may include one or more adjacent BPUs that have already been encoded (in the forward path) and reconstructed (in the reconstruction path) in the same picture. The encoder can generate the predicted BPU 208 by extrapolating the adjacent BPUs. Extrapolation techniques may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, etc. In some embodiments, the encoder may perform extrapolation at the pixel level, for example by extrapolating the value of a corresponding pixel for each pixel of the predicted BPU 208. The adjacent BPUs used for extrapolation may be positioned relative to the original BPU from various directions, such as in the vertical direction (e.g., above the original BPU), the horizontal direction (e.g., to the left of the original BPU), the diagonal direction (e.g., lower left, lower right, upper left, or upper right of the original BPU), or any direction defined in the video coding standard used. For intra-frame prediction, prediction data 206 may include, for example, the location (e.g., coordinates) of the neighboring BPU used, the size of the neighboring BPU used, extrapolation parameters, and the orientation of the neighboring BPU used relative to the original BPU.

[0067] In another example, during the temporal prediction phase 2044, the encoder can perform inter-frame prediction. For the original BPU of the current image, the prediction reference 224 can include one or more images (referred to as "reference images") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference image can be encoded and reconstructed using the BPU. For example, the encoder can add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a reconstructed BPU. When all reconstructed BPUs of the same image are generated, the encoder can generate the reconstructed image as the reference image. The encoder can perform a "motion estimation" operation to search for matching regions within the range of the reference image (referred to as a "search window"). The position of the search window in the reference image can be determined based on the position of the original BPU in the current image. For example, the search window can be centered in the reference image at a position with the same coordinates as the original BPU in the current image and can extend outward by a predetermined distance. When the encoder identifies a region similar to the original BPU in the search window (e.g., by using a pixel recursive algorithm, block matching algorithm, etc.), the encoder can determine such a region as a matching region. The matching region can have a different size than the original BPU (e.g., less than, equal to, greater than, or different shape). This is because the reference image and the current image are temporarily separated in the timeline (e.g., as...). Figure 1 As shown in the image, it can be assumed that over time, the matching region "moves" to the original BPU's location. The encoder can record the direction and distance of this movement as a "motion vector" when using multiple reference images (e.g., such as...). Figure 1 When working with image 106, the encoder can search for matching regions and determine the associated motion vector for each reference image. In some embodiments, the encoder can assign weights to the pixel values ​​of the matching regions of each matching reference image.

[0068] Motion estimation can be used to identify various types of motion, such as translation, rotation, scaling, etc. For inter-frame prediction, prediction data 206 may include, for example, the location (e.g., coordinates) of the matching region, the motion vector associated with the matching region, the number of reference images, the weights associated with the reference images, etc.

[0069] To generate the predicted BPU 208, the encoder can perform a "motion compensation" operation. Motion compensation can be used to reconstruct the predicted BPU 208 based on the predicted data 206 (e.g., motion vectors) and the prediction reference 224. For example, the encoder can move the matching region of the reference image according to the motion vectors, where the encoder can predict the original BPU of the current image. When using multiple reference images (e.g., such as...), Figure 1In image 106), the encoder can move the matching region of the reference image based on the corresponding motion vector and average pixel value of the matching region. In some embodiments, if the encoder has already assigned weights to the pixel values ​​of the matching regions of each matching reference image, the encoder can add a weighted sum of the pixel values ​​of the moved matching regions.

[0070] In some embodiments, inter-frame prediction can be unidirectional or bidirectional. Unidirectional inter-frame prediction can use one or more reference images in the same temporal direction as the current image. For example, Figure 1 Image 104 in the example is a one-way inter-frame prediction image, where the reference image (i.e., image 102) precedes image 104. Two-way inter-frame prediction can use one or more reference images in two temporal directions relative to the current image. For example, Figure 1 Image 106 in the image is a bidirectional inter-frame prediction image, in which the reference images (i.e., images 104 and 108) are in two time directions relative to image 104.

[0071] Referring again to the forward path of process 200B, after spatial prediction stage 2042 and temporal prediction stage 2044, in mode determination stage 230, the encoder can select a prediction mode (e.g., one of intra-frame prediction or inter-frame prediction) for the current iteration of process 200B. For example, the encoder can perform rate-distortion optimization techniques, whereby the encoder selects a prediction mode based on the bit rate of candidate prediction modes and the distortion of the reference image reconstructed under the candidate prediction modes, in order to minimize the value of the cost function. Based on the selected prediction mode, the encoder can generate the corresponding prediction BPU 208 and prediction data 206.

[0072] In the reconstruction path of process 200B, if intra-frame prediction mode is selected in the forward path, the encoder can directly input prediction reference 224 (e.g., the current BPU that has been encoded and reconstructed in the current image) into spatial prediction stage 2042 for later use (e.g., for extrapolating the next BPU of the current image) after generating prediction reference 224. If inter-frame prediction mode has been selected in the forward path, the encoder can input prediction reference 224 (e.g., the current image where all BPUs have been encoded and reconstructed) into loop filter stage 232 after generating prediction reference 224. At this time, the encoder can apply loop filters to prediction reference 224 to reduce or eliminate distortions (e.g., blockiness) introduced by inter-frame prediction. The encoder can apply various loop filter techniques in loop filter stage 232, such as deblocking, sample adaptive offset, adaptive loop filter, etc. The loop-filtered reference image can be stored in buffer 234 (or "decoded image buffer") for later use (e.g., as an inter-frame prediction reference image for future images of video sequence 202). The encoder may store one or more reference images in buffer 234 for use in the time prediction stage 2044. In some embodiments, the encoder may encode parameters of the loop filter (e.g., loop filter strength), as well as quantization transform coefficients 216, prediction data 206, and other information in the binary encoding stage 226.

[0073] Figure 3A A schematic diagram of an example decoding process 300A consistent with embodiments of this application is shown. Process 300A may be a decompression process corresponding to compression process 200A in FIG. 2. In some embodiments, process 300A may be similar to the reconstruction path of process 200A. The decoder can decode the video bitstream 228 into video stream 304 according to process 300A. Video stream 304 may be very similar to video sequence 202. However, due to information loss during compression and decompression (e.g., Figure 2A-2B In the quantization stage 214), the video stream 304 is typically different from the video sequence 202. Similar to... Figure 2A-2B In processes 200A and 200B, the decoder can perform process 300A on each image encoded in the video stream 228 at the Basic Processing Unit (BPU) level. For example, the decoder can perform process 300A iteratively, where the decoder can decode the Basic Processing Unit in one iteration of decoding process 300A. In some embodiments, the decoder can perform process 300A in parallel on a region (e.g., region 114-118) of each image encoded in the video stream 228.

[0074] In Figure A, the decoder may input a portion of the video bitstream 228 associated with a basic processing unit (referred to as an "encoded BPU") of the encoded image into binary decoding stage 302. In binary decoding stage 302, the decoder may decode this portion into prediction data 206 and quantization transform coefficients 216. The decoder may input the quantization transform coefficients 216 into inverse quantization stage 218 and inverse transform stage 220 to generate a reconstructed residual BPU 222. The decoder may input the prediction data 206 into prediction stage 204 to generate a predicted BPU 208. The decoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a prediction reference 224. In some embodiments, the prediction reference 224 may be stored in a buffer (e.g., a decoded image buffer in computer memory). The decoder may input the prediction reference 224 into prediction stage 204 for performing a prediction operation in the next iteration of process 300A.

[0075] The decoder can iteratively execute process 300A to decode each encoded BPU of the encoded image and generate a prediction reference 224 for encoding the next encoded BPU of the encoded image. After decoding all encoded BPUs of the encoded image, the decoder can output the image to video stream 304 for display and continue decoding the next encoded image in video stream 228.

[0076] In binary decoding stage 302, the decoder can perform the inverse operation of the binary encoding technique used by the encoder (e.g., entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in addition to the prediction data 206 and quantization transform coefficients 216, the decoder can also decode other information in binary decoding stage 302, such as the prediction mode, parameters of the prediction operation, transform type, quantization parameter process (e.g., quantization parameters), encoder control parameters (e.g., bit rate control parameters), etc. In some embodiments, if the video stream 228 is transmitted in packets over a network, the decoder can unpack it before inputting the video stream 228 into binary decoding stage 302.

[0077] Figure 3B A schematic diagram of another example decoding process 300B consistent with embodiments of this application is shown. Process 300B can be modified from process 300A. For example, process 300B can be used by a decoder conforming to a hybrid video coding standard (e.g., H.26x series). Compared to process 300A, process 300B further divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and further includes a loop filter stage 232 and a buffer 234.

[0078] In process 300B, for the encoding basic processing unit (referred to as the "current BPU") of the encoded picture being decoded (referred to as the "current picture"), the prediction data 206 decoded by the decoder from the binary decoding level 302 can include various types of data depending on the prediction mode used by the encoder to encode the current BPU. For example, if the encoder uses intra-frame prediction to encode the current BPU, the prediction data 206 can include prediction mode indicators (e.g., flag values) indicating intra-frame prediction, parameters for the intra-frame prediction operation, etc. Parameters for the intra-frame prediction operation can include, for example, the positions (e.g., coordinates) of one or more neighboring BPUs used as references, the sizes of neighboring BPUs, extrapolation parameters, the orientations of neighboring BPUs relative to the original BPU, etc. For example, if the encoder uses inter-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating inter-frame prediction, parameters of the inter-frame prediction operation, etc., which may include, for example, the number of reference images associated with the current BPU, the weights associated with the reference images respectively, the positions (e.g., coordinates) of one or more matching regions in the corresponding reference images, one or more motion vectors associated with the matching regions respectively, etc.

[0079] Based on the prediction mode indicator, the decoder can decide whether to perform spatial prediction (e.g., intra-frame prediction) in the spatial prediction phase 2042 or temporal prediction (e.g., inter-frame prediction) in the temporal prediction phase 2044. The details of performing this spatial or temporal prediction are detailed in... Figure 2B As described above, it will not be repeated below. After performing this spatial or temporal prediction, the decoder can generate the predicted BPU 208. (As...) Figure 3A As described, the decoder can add the predicted BPU 208 and the reconstructed residual BPU 222 to generate the prediction reference 224.

[0080] In process 300B, the decoder can input prediction reference 224 to spatial prediction stage 2042 or temporal prediction stage 2044 to perform prediction operations in the next iteration of process 300B. For example, if the current BPU is decoded using intra-frame prediction in spatial prediction stage 2042, the decoder can directly input prediction reference 224 to spatial prediction stage 2042 for later use (e.g., for extrapolating the next BPU of the current image) after generating prediction reference 224 (e.g., the decoded current BPU). If the current BPU is decoded using inter-frame prediction in temporal prediction stage 2044, the encoder can input prediction reference 224 to loop filter stage 232 after generating prediction reference 224 (e.g., a reference image where all BPUs have been decoded) to reduce or eliminate distortion (e.g., block artifacts). The decoder can... Figure 2B The loop filter is applied to prediction reference 224 in the manner described herein. The loop-filtered reference image can be stored in buffer 234 (e.g., a decoded image buffer in computer memory) for later use (e.g., as an inter-frame prediction reference image used as a future encoded image for video bitstream 228). The decoder can store one or more reference images in buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the prediction data can further include parameters of the loop filter (e.g., loop filter strength) when the prediction mode indicator of the prediction data 206 indicates that inter-frame prediction is used to encode the current BPU.

[0081] Figure 4A This is a block diagram of an example apparatus 400 for encoding or decoding video, consistent with embodiments of this application. Figure 4A As shown, device 400 may include processor 402. When processor 402 executes the instructions described herein, device 400 may become a dedicated machine for video encoding or decoding. Processor 402 may be any type of circuit capable of manipulating or processing information. For example, processor 402 may include any number of central processing units (or “CPU”), graphics processing units (or “GPU”), neural processing units (“NPU”), microcontroller units (“MCU”), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLAs), programmable array logic (PALs), general-purpose array logic (GALs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), system-on-a-chip (SoCs), application-specific integrated circuits (ASICs), etc. In some embodiments, processor 402 may also be a group of processors grouped into a single logic controller. For example, as Figure 4A As shown, processor 402 may include multiple processors, including processor 402a, processor 402b and processor 402n.

[0082] Device 400 may also include memory 404 configured to store data (e.g., a set of instructions, computer code, intermediate data, etc.). For example, as Figure 4AAs shown, the stored data may include program instructions (e.g., program instructions for implementing stages in processes 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequence 202, video stream 228, or video stream 304). Processor 402 can access the program instructions and data for processing (e.g., via bus 410) and execute program instructions to manipulate or control the data for processing. Memory 404 may include a high-speed random access memory device or a non-volatile memory device. In some embodiments, memory 404 may include any number of random access memories (RAM), read-only memories (ROM), optical discs, magnetic disks, hard disks, solid-state drives, flash drives, secure digital cards (SD cards), memory sticks, compact flash (CF) cards, etc. Memory 404 may also be a group of memories grouped into single logical components. Figure 4A (Not shown in the image).

[0083] Bus 410 may be a communication device for transmitting data between components within device 400, such as an internal bus (e.g., CPU-memory bus), an external bus (e.g., a Universal Serial Bus port, a Peripheral Component Interconnect Fast Port), etc.

[0084] For ease of explanation and to avoid ambiguity, the processor 402 and other data processing circuits are collectively referred to as "data processing circuits" in this application. The data processing circuits may be implemented entirely as hardware, a combination of software and hardware, or firmware. Furthermore, the data processing circuits may be a single, independent module, or may be wholly or partially integrated into any other component of the device 400.

[0085] Device 400 may also include a network interface 406 to provide wired or wireless communication in relation to a network (e.g., the Internet, intranet, local area network, mobile communication network, etc.). In some embodiments, network interface 406 may include any number of network interface controllers (NICs), radio frequency (RF) modules, repeaters, transceivers, modems, routers, gateways, any combination of wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (“NFC”) adapters, cellular network chips, etc.

[0086] In some embodiments, optionally, the device 400 may further include a peripheral interface 408 to provide connectivity to one or more peripheral devices. Figure 4A As shown, peripheral devices may include, but are not limited to, cursor control devices (such as mice, touchpads, or touchscreens), keyboards, displays (such as cathode ray tube displays, liquid crystal displays, or light-emitting diode displays), video input devices (such as cameras or input interfaces coupled to video files), etc.

[0087] It should be noted that the video codec (e.g., the codec for executing processes 200A, 200B, 300A, or 300B) can be implemented as any combination of any software or hardware modules in device 400. For example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more software modules of device 400, such as program instructions that can be loaded into memory 404. As another example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more hardware modules of device 400, such as dedicated data processing circuitry (e.g., FPGA, ASIC, NPU, etc.).

[0088] For a CU using inter-frame predictive coding, a reference block in a previously decoded image (i.e., a reference image) is identified as a predictor. The relative position between the reference block in the reference image and the coded block in the current image is defined as a motion vector (MV). The motion information of the current CU is specified by the number of predictor variables, the reference image index, and the corresponding MV. After obtaining the prediction based on the motion information through motion compensation, the residual between the predicted signal and the original signal can be further transformed, quantized, and entropy-coded before being packaged into the output bitstream.

[0089] In some cases, motion information of the current CU can be predicted using motion information from its spatiotemporally adjacent CUs. A merging pattern can be employed to reduce the number of bits encoded for the motion information. In this merging pattern, the motion information is derived from spatially or temporally adjacent blocks, and a tagged merging index indicates from which adjacent block the motion information was derived.

[0090] In HEVC, a merge candidate list can be built based on the following candidates:

[0091] (1) A maximum of four spatial merging candidates originating from five spatially adjacent blocks;

[0092] (2) A time-merging candidate derived from a time-coordinated block;

[0093] (3) Additional merging candidates, including combined bidirectional prediction candidates and zero motion vector candidates.

[0094] The first candidate in the merge candidate list is spatial neighbor. Figure 4BThe locations of five spatial candidates are displayed. The availability of each candidate location is checked according to the order {A1, B1, B0, A0, B2}. If a spatially adjacent block is predicted intra-frame, or if the location is outside the current segment or fragment, it can be considered an unavailable merge candidate (merge candidate block). Furthermore, some redundancy checks can be performed to ensure that motion data from adjacent blocks is as unique as possible. To reduce the complexity caused by redundancy checks, only a limited number of redundancy checks can be performed, and uniqueness may not always be guaranteed. For example, given the order {A1, B1, B0, A0, B2}, B0 only checks B1, A0 only checks A1, and B2 only checks A1 and B1.

[0095] like Figure 4C As shown, for time-merging candidates, if available, the bottom-right position C0 outside the juxtaposition block of the reference images is used. Otherwise, the center position C1 can be used instead. The index marked in the fragment header indicates which list of reference images is used for juxtaposition. For example... Figure 5 As shown, the MV of the colocation block can be scaled based on the difference in Picture Order Count (POC) before being inserted into the merge list.

[0096] The maximum number C of merge candidates can be specified in the fragment header. If the number of available merge candidates (including the time candidates) is greater than C, only the first C-1 spatial and time candidates are retained. Otherwise, if the number of available merge candidates is less than C, additional candidates are generated until the number equals C. Because the ability to parse encoded data is independent of the number of available merge candidates, parsing can be simplified and made more robust. In the Common Test Conditions (CTC), the maximum number C of merge candidates is set to 5.

[0097] For segment B, additional merge candidates are generated by combining two available candidates according to the predefined order of reference image lists 0 and 1. For example, the first generated candidate uses the first merge candidate from list 0 and the second merge candidate from list 1. HEVC specifies a total of 12 predefined pairs of motion vectors in the constructed merge candidate list, in the order (0,1), (1,0), (0,2), (2,1), (0,3), (3,0), (1,3), (3,1), (2,3), (3,2), (3,2), (0,2), (0,3), (1,3), (3,1), (2,3), (3,2), (0,2), (0,3), (1,3), (3,1), (2,3), (3,2), where (i,j) represents the index of the available merge candidate. After removing redundant entries, a maximum of five candidates can be included.

[0098] When the fragment is a P-fragment or the number of merge candidates is still less than C, the zero motion vector associated with the reference index (from 0 to the number of reference images minus 1) is used to fill any remaining entries in the merge candidate list.

[0099] In VVC, the merge candidate list is constructed sequentially using the following five candidates:

[0100] Candidates for spatial merging from spatially adjacent CUs;

[0101] Time merging candidates from co-located CUs;

[0102] History-based motion vector predictor (HMVP) from FIFO table;

[0103] Paired average candidates; and

[0104] Zero MV.

[0105] The definitions of spatial merge candidates and temporal merge candidates are the same as those in HEVC. After the spatial and temporal merge candidates, HMVP merge candidates are added to the merge list. In HMVP, motion information of the previously encoded block is stored in a table and used as the motion vector prediction factor for the current CU. This table, containing multiple HMVP candidates, is maintained during encoding / decoding. The table is reset (cleared) when a new CTU row is encountered. When a non-sub-block inter-coded CU exists, the associated motion information is added as a new HMVP candidate to the last item in the table.

[0106] In VVC, the HMVP table size can be set to 6, meaning a maximum of 6 HMVP candidates can be added to the table. When inserting a new motion candidate into the table, a constraint-based First-In-First-Out (FIFO) rule can be used, where a redundancy check is first applied to see if a duplicate HMVP exists in the table. If found, the duplicate HMVP is removed from the table, and then all HMVP candidates can be moved forward.

[0107] During the construction of the merge candidate list, the latest HMVP candidates in the table are checked sequentially and inserted into the merge candidate list, following the TMVP candidates. Redundancy checks can be used to examine the HMVP candidates with spatial or temporal merge candidates.

[0108] After inserting HMVP candidates, if the merge candidate list is not yet full, pairwise average candidates are added. Pairwise average candidates are generated by averaging predefined candidate pairs in the existing merge candidate list. The predefined candidate pairs are defined as {(0,1),(0,2),(1,2),(0,3),(1,3),(2,3)}, where the numbers represent the merge index in the merge candidate list. The average motion vector is calculated separately for each list of reference images. If two motion vectors are available in a list, they are averaged even if they point to different reference images. If only one motion vector is available, the available motion vector is used directly. If no motion vector is available, the list is considered invalid.

[0109] If the merge list is still not full after adding pairwise average merge candidates, insert a zero motion vector at the end until the maximum number of merge candidates is reached.

[0110] In VVC, in addition to the regular merging modes, the construction of the merging candidate list can also be used for merging modes of Motion Vector Difference (MMVD) and Triangle Partitioning (TPM).

[0111] In MMVD, candidate phases are first selected from the candidate merging list, and then further refined using signal motion vector difference (MVD) information. The size of the MMVD candidate merging list is set to 2. Candidate merging flags can be used to specify which of the two MMVD candidates is used as the reference motion vector (MV). MVD information can be represented by distance and direction indices. The distance index specifies motion amplitude information and indicates a preset offset from the reference motion vector MV. The relationship between the distance index and the preset offset is as follows: Figure 6 As shown. The direction index specifies the sign of the offset added to the reference motion vector MV; for example, 0 indicates a positive sign and 1 indicates a negative sign.

[0112] In TPM, the CU is uniformly divided into two triangular partitions using diagonal or anti-diagonal segmentation. Each triangular partition in the CU can be used for inter-frame prediction using its own motion. Only a single prediction is allowed per partition. That is, each partition has one motion vector and one reference index. A single prediction motion constraint is applied to ensure that, similar to bidirectional prediction, each CU requires only two motion-compensated predictions. If the current CU uses the triangular partitioning mode, a flag can be further used to indicate the orientation of the triangular partition (diagonal or anti-diagonal), and two merge indices (one per partition). After predicting each triangular partition, a blending process with adaptive weights is used to adjust the sample values ​​along the diagonal or anti-diagonal edges. This corresponds to the predicted signal for the entire CU, and the transform and quantization process can be applied to the entire CU as in regular inter-frame mode. A merge candidate list can be constructed. The maximum number of TPM merge candidates can be explicitly stated in the segment header and set to 5 in the CTC.

[0113] In VVC, a merge candidate list is constructed, including spatial candidates, temporal candidates, HMVP, and pairwise average candidates. Different merge list sizes are applied for different inter-frame modes. For example, spatial merge candidates can be inserted into the merge list in the order {A1, B1, B0, A0, B2}. However, the process of constructing spatial merge candidates has not changed from HEVC to VVC; it does not take into account new merge candidates (such as HMVP and pairwise average candidates) and new inter-frame modes (such as MMVD and TPM) in VVC. This leads to various shortcomings of the current spatial merge candidate list.

[0114] For example, the order of spatial merge candidates can be improved. The number of spatial merge candidates can be adjusted. Furthermore, the construction of spatial merge candidates is fixed for regular mode, MMVD, and TPM modes, which limits the potential of the merging method. Additionally, the construction of spatial merge candidates is fixed for both low-latency and non-low-latency images, reducing flexibility. To address the above and other issues, this application provides various solutions.

[0115] For example, in some embodiments, the order of spatial merge candidates can be changed. A new spatial merge candidate sequence {B1, A1, B0, A0, B2} can be applied. The positions of spatially adjacent blocks B1, A1, B0, A0, and B2 are as follows: Figure 7 As shown. In some embodiments, this order can be changed to {B1, A1, B0, A0, B2}.

[0116] The new order corresponds sequentially to the upper adjacent block, left adjacent block, upper adjacent block, left adjacent block, and upper left adjacent block, alternating between the upper and left adjacent blocks. Furthermore, the new spatial merging candidate order can achieve higher coding performance. For example... Figure 8 and Figure 9 As shown. According to some embodiments, the proposed method can achieve an average coding gain of 0.05% and 0.21% in random access (RA) and low latency (LD) configurations, respectively, compared to VTM-6.

[0117] In some embodiments, the number of spatial merge candidates can be varied. To achieve a better trade-off between computational complexity and coding performance, a reduction in the number of spatial merge candidates is proposed and applied in various embodiments of this application. When the number of spatial merge candidates is limited to 2, a build order {B1, A1} can be applied. For example, if a neighboring block B1 is available, it can be selected and inserted into the merge list. Then, if a neighboring block A1 is available and is different from B1, it can be selected and inserted into the merge list. After inserting the spatial merge candidate {B1, A1}, the following TMVP, HMVP, and pairwise average candidates can be added to the merge list.

[0118] When the number of space merge candidates is limited to 3, the build order {B1, A1, B0} can be applied. The checking order for adjacent blocks is B1->A1->B0, and if the corresponding MV is available and not redundant, it can be inserted into the merge list.

[0119] When using the spatial merging candidate {B1, A1, B0}, the experimental results compared to VTM-6 are as follows: Figure 10 and Figure 11 As shown. According to some embodiments, the proposed technique can achieve coding gains of 0.00% and 0.10% in RA and LD configurations.

[0120] Furthermore, in some VVC techniques, the total number of merge candidates can be marked. In some embodiments of this disclosure, it is suggested that the number of spatial merge candidates be additionally marked to provide greater flexibility in constructing the merge candidate list. Considering the prediction structure of the current image, the number of spatial merge candidates can be set to different values. If the current image is a non-low-latency image, the number of spatial merge candidates can be set to a first value. A non-low-latency image can refer to an image encoded according to display order, using reference images from the past and future. Otherwise, if the current image is a low-latency image, the number of spatial merge candidates can be set to a second value. A low-latency image can refer to an image encoded according to display order, using only past reference images. The first value can be greater than the second value. The first and second values ​​can be explicitly marked in the bitstream, for example, in the fragment header. Figure 12 An example is shown in the figure.

[0121] Syntax element num_spatial_merge_cand_minus2 (e.g., Figure 12 The element 1201 in the table indicates the number of spatial merge candidates for the current fragment. The value of num_spatial_merge_cand_minus2 can be in the range of 0 to 3, inclusive. When the element num_spatial_merge_cand_minus2 does not exist, it can be inferred that it is equal to 0.

[0122] Based on the reference image used to encode the current segment, the segment can be classified as low-latency or non-low-latency, and different numbers of merge candidates can be used. The value of num_spatial_merge_cand_minus2 can be set accordingly by the encoder and sent in the bitstream.

[0123] Alternatively, instead of marking a syntax element in the fragment header, the two syntax elements num_spatial_merge_cand_minus2_non_lowdelay and num_spatial_merge_cand_minus2_lowdelay can be marked in the Picture Parameter Set (PPS) or Sequence Parameter Set (SPS), such as... Figure 13 As shown, Figure 13 (For example, element 1301) and Figure 14 (For example, element 1401). Furthermore, at the fragment level, depending on the fragment type, a corresponding number of space-merging candidates can be used.

[0124] The values ​​of `num_spatial_merge_cand_minus2_non_lowdelay` and `num_spatial_merge_cand_minus2_lowdelay` represent the number of spatial merging candidates for non-low-latency segments and low-latency segments, respectively. The values ​​of `num_spatial_merge_cand_minus2_non_lowdelay` and `num_spatial_merge_cand_minus2_lowdelay` can range from 0 to 3, inclusive. If either `num_spatial_merge_cand_minus2_non_lowdelay` or `num_spatial_merge_cand_minus2_lowdelay` is absent, it can be inferred that it is equal to 0.

[0125] In some embodiments, the individual build order of spatial merge candidates can be applied to different inter-frame modes. For example, two build orders for spatial merge candidates can be considered, including {B1, A1, B0, A0, B2} and {A1, B1, B0, A0, B2}. Different build orders can be used for regular merge mode, MMVD mode, and TPM mode. In some embodiments, it is recommended to use {B1, A1, B0, A0, B2} for regular merge mode and TPM mode, and {A1, B1, B0, A0, B2} for MMVD mode. Experimental results of exemplary embodiments are as follows: Figure 15 and Figure 16 As shown in the figure. The results indicate that the average coding gain of this method is 0.07% and 0.16% in RA and LD configurations, respectively.

[0126] Based on this application, those skilled in the art will understand that other combinations of spatial merge candidate order and merge mode can be used. For example, {B1, A1, B0, A0, B2} can be used only in the regular merge mode, while {A1, B1, B0, A0, B2} can be used in both MMVD and TMP modes.

[0127] In some embodiments, an adaptive construction order of spatial merge candidates can be applied based on frame type. For example, different spatial merge candidate construction methods can be applied to different types of inter-frame coded images, such as low-latency and non-low-latency images. In some embodiments, for low-latency images, the construction order of spatial merge candidates {B1, A1, B0, A0, B2} can be used in regular merge mode, TPM mode, and MMVD mode. For non-low-latency images, the construction order of spatial merge candidates {B1, A1, B0, A0, B2} can be used in regular merge mode and TPM mode, and the construction order of spatial merge candidates {A1, B1, B0, A0, B2} can be used in MMVD mode. Experimental results of exemplary embodiments are as follows: Figure 17 and Figure 18 As shown in the figure. The results indicate that the average coding gain of this method is 0.08% and 0.21% under RA and LD configurations, respectively.

[0128] Figure 19 A flowchart of an exemplary video processing method 1900 consistent with embodiments of this disclosure is illustrated. In some embodiments, method 1900 may be comprised of an encoder, a decoder, or a device (e.g., Figure 4A The device (400) is executed by one or more software or hardware components. For example, a processor (e.g., Figure 4A The processor 402) can execute method 1900. In some embodiments, method 1900 can be implemented by a computer program product contained in a computer-readable medium, including a processor (e.g., a computer) that executes method 1900. Figure 4A The device 400 executes computer-executable instructions, such as program code.

[0129] In step 802, a set of spatial merging candidates can be inserted into the merging candidate list of the coded block, for example, by an encoder, decoder, or device (e.g., a device). Figure 4A (400). In VVC, the order of space merge candidates can be improved. Space merge candidates can be inserted in the following order: top adjacent block, left adjacent block, top adjacent block, left adjacent block, and top-left adjacent block. For example, Figure 7 The new order of spatial merging candidates {B1, A1, B0, A0, B2} is shown in the figure.

[0130] The number of space merging candidates can be adjusted. In step 804, a preset limit on the number of space merging candidates is determined.

[0131] In step 806, if the number is limited to 2, spatial merge candidates are inserted into the merge candidate list in the following order: top adjacent block, left adjacent block. When the number of spatial merge candidates is limited to 2, the build order {B1, A1} can be applied. For example, if adjacent block B1 is available, it can be selected and inserted into the merge list. Then, if adjacent block A1 is available and different from B1, it can be selected and inserted into the merge list. After inserting the spatial merge candidate {B1, A1}, the following TMVP, HMVP, and pairwise average candidates can be added to the merge list.

[0132] In step 808, if the number is limited to 3, spatial merge candidates are inserted into the merge candidate list in the following order: top adjacent block, left adjacent block, top adjacent block. When the number of spatial merge candidates is limited to 3, the build order {B1, A1, B0} can be applied. The adjacent block checking order is B1->A1->B0, and if available and not redundant, the corresponding MV can be inserted into the merge list.

[0133] In some embodiments, at least one of the following may be added to the merging candidate list: a time merging candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

[0134] In HMVP, motion information from previously encoded blocks is stored in a FIFO table and used as a motion vector predictor for the current coding unit. A table with multiple HMVP candidates is maintained during encoding / decoding. This table is reset (cleared) when a new CTU row is encountered. When a non-sub-block inter-coding unit exists, the motion information associated with that non-sub-block inter-coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0135] The pairwise averaged candidate is generated by averaging the candidate pairs in the merge candidate list, and is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0136] If the merge list is not full after adding pairwise average merge candidates, insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0137] In step 810, the encoder or decoder determines whether to apply a first encoding mode or a second encoding mode to the encoded block. The first encoding mode differs from the second encoding mode. In some embodiments, each of the first and second encoding modes may be one of a regular merging mode, a merging mode with motion vector difference (MMVD), and a triangular partitioning mode (TPM).

[0138] In step 812, when the first coding mode is applied to the coding block, spatial merging candidates are inserted according to the first construction order. For example, in MMVD, a merging candidate is first selected from the merging candidate list, then refined using marked motion vector difference (MVD) information, and a merging candidate flag is used to specify which of the two MMVD candidates will be used as the base motion vector. MVD information can be represented by a distance index and a direction index. The distance index specifies motion amplitude information and indicates a predefined offset from the base MV. The relationship between the distance index and the preset offset is as follows: Figure 6 As shown in the example, the direction index specifies the sign of the offset added to the base MV; for example, 0 indicates a positive sign and 1 indicates a negative sign.

[0139] In step 814, when the second coding mode is applied to the coding block, spatial merging candidates are inserted according to the second construction order. For example, in TPM, the coding unit is uniformly divided into two triangular partitions using at least one of diagonal partitioning or anti-diagonal partitioning. Each triangular partition in the CU can perform inter-frame prediction using its own motion. Only a single prediction is allowed per partition. That is, each partition has a motion vector and a reference index.

[0140] In step 816, it is determined whether the coded block is part of a low-latency image or part of a non-low-latency image.

[0141] In step 818, when the encoded block is part of a low-latency image, spatial merge candidate merges are inserted according to a third build order. In some embodiments, for low-latency images, the build order of the spatial merge candidates {B1, A1, B0, A0, B2} can be used in regular merge mode, TPM mode, and MMVD mode.

[0142] In step 820, when the encoded block is part of a non-low-latency image, spatial merge candidates are inserted according to a fourth build order. The third and fourth build orders differ from the fourth build order. The third and fourth build orders are used for merge modes with motion vector differences. In some embodiments, for non-low-latency images, the build order of the spatial merge candidates {B1, A1, B0, A0, B2} can be used in both regular merge mode and TPM mode, and the spatial merge candidates with the build order {A1, B1, B0, A0, B2} can be used in MMVD mode.

[0143] Consistent with this application, those skilled in the art will understand that the above-described methods can be used in combination or individually. For example, the technique of reducing the number of spatial merging candidates can be used in conjunction with the proposed method of using spatial merging candidates with different construction orders for different patterns.

[0144] In some embodiments, a non-transitory computer-readable storage medium including instructions that can be executed by a device for performing the methods described above (e.g., the encoders and decoders disclosed herein). Common forms of non-transitory media include, for example, floppy disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs or any other flash memory, NVRAM, caches, registers, any other memory chips or cassettes, and the same network versions. The device may include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memory.

[0145] It should be noted that the relational terms such as "first" and "second" used in this document are used only to distinguish one entity or operation from another, and do not require or imply any actual relationship or order between these entities or operations. Furthermore, words such as "contains," "has," "includes," and "includes," as well as other similar forms, have the same meaning and are open-ended, as one or more items following any of these words are not intended to be an exhaustive list of such items, or limited to the listed items.

[0146] As used herein, unless otherwise expressly stated, the term "or" covers all possible combinations unless impractical. For example, if a component is declared to include A or B, then unless otherwise expressly stated or impractical, the component may include A, or B, or A and B. As a second example, if a component is declared to include A, B, or C, then unless otherwise expressly stated or impractical, the component may include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0147] It is understood that the above embodiments can be implemented by hardware or software (program code) or a combination of hardware and software. If implemented by software, it can be stored in the above-described computer-readable medium. When executed by a processor, the software can perform the disclosed methods. The computing units and other functional units described in this application can be implemented by hardware, by software, or by a combination of hardware and software. It will also be understood by those skilled in the art that the above-described multiple modules / units can be combined into one module / unit, and each of the above-described modules / units can be further divided into multiple sub-modules / sub-units.

[0148] The above embodiments may be further described using the following terms:

[0149] 1. A video processing method, comprising:

[0150] Insert a set of spatial merging candidates into the merging candidate list of the coded blocks.

[0151] The space merging candidates are inserted in the following order: upper adjacent block, left adjacent block, upper adjacent block, left adjacent block and upper left adjacent block.

[0152] 2. The method described in Clause 1 further includes:

[0153] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

[0154] 3. The method according to Clause 2, wherein the motion information of the prior encoded block is stored in the FIFO table and used as a motion vector predictor for the current encoded unit.

[0155] 4. The method according to any one of Clauses 2 and 3, wherein motion information associated with a non-sub-block inter-coding unit is added to the last entry of the FIFO table as a new HMVP candidate.

[0156] 5. The method according to Clause 2, wherein,

[0157] The pairwise averaged candidates are generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidates are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0158] 6. The method according to Clause 2, wherein,

[0159] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0160] 7. A video processing method, comprising:

[0161] Based on the preset limit on the number of spatial merging candidates, a set of spatial merging candidates are inserted into the merging candidate list of the coding block.

[0162] If the quantity is limited to 2, then the group of space merging candidates are inserted into the merging candidate list in the following order: upper adjacent block, left adjacent block; and

[0163] If the number is limited to 3, the space merging candidates are inserted into the merging candidate list in the following order: upper adjacent block, left adjacent block, upper adjacent block.

[0164] 8. The method described in Clause 7, wherein

[0165] In response to encoding the current image according to the display order, using past reference images and future reference images, the number of spatial merging candidates is set to a first value; and

[0166] In response to encoding the current image using past reference images according to the display order, the number of spatial merging candidates is set to a second value that is less than the first value.

[0167] 9. The method described under Clause 7 further includes:

[0168] Mark the number of space merge candidates inserted into the merge candidate list.

[0169] 10. The method according to any one of clauses 7-9, further comprising:

[0170] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the FIFO table, a pairwise average candidate, or a zero motion vector.

[0171] 11. The method according to Clause 10, wherein,

[0172] Motion information of the previously encoded blocks is stored in a FIFO table and used as a motion vector predictor for the current encoding unit.

[0173] 12. The method according to any one of clauses 10 and 11, wherein

[0174] Motion information associated with coding units that are not sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP candidate.

[0175] 13. The method according to Clause 10, wherein,

[0176] The pairwise averaged candidate is generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidate is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0177] 14. The method according to Clause 10, wherein,

[0178] The zero motion vector is inserted at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0179] 15. A video processing method, comprising:

[0180] A set of spatial merging candidates is inserted into the merging candidate list of the coded blocks, where:

[0181] When the first encoding mode is applied to the encoded block, the space merging candidates are inserted according to the first construction order; and

[0182] When the second encoding mode is applied to the encoded block, the space merging candidates are inserted according to the second construction order.

[0183] The first build order is different from the second build order.

[0184] 16. The method according to Clause 15, wherein the first encoding mode and the second encoding mode are two different modes selected from the following:

[0185] The modes include standard merging mode, motion vector difference merging mode (MMVD), and triangular partitioning mode (TPM).

[0186] 17. The method according to Clause 16, wherein,

[0187] In MMVD, a candidate for merging is first selected from the list of candidates for merging, then refined using the marked motion vector difference (MVD) information, and a merging candidate flag is marked to specify which of the two MMVD candidates to use as the base motion vector.

[0188] 18. The method according to Clause 16, wherein,

[0189] In TPM, the coding unit is uniformly divided into two triangular partitions using at least one of diagonal partitioning or anti-diagonal partitioning.

[0190] 19. The method according to any one of clauses 15 and 16 further comprises:

[0191] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor from the FIFO table, a pairwise average candidate, or a zero motion vector.

[0192] 20. The method according to Clause 19, wherein,

[0193] The motion information of the previously encoded blocks is stored in a FIFO table and used as the motion vector predictor for the current encoding unit.

[0194] 21. The method according to any one of clauses 19 and 20, wherein

[0195] Motion information associated with coding units that are not sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP candidate.

[0196] 22. The method according to Clause 19, wherein,

[0197] The pairwise averaged candidates are generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidates are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0198] 23. The method according to Clause 19, wherein,

[0199] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0200] 24. A video processing method, comprising:

[0201] A set of spatial merging candidates is inserted into the merging candidate list of the coded blocks, where:

[0202] When the encoded block is part of a low-latency image, insert the group of spatial merge candidates according to the first build order; and

[0203] When the encoded block is part of a non-low-latency image, the group of spatial merging candidates is inserted according to the second build order.

[0204] The first build order is different from the second build order.

[0205] 25. The method described according to Clause 24, wherein,

[0206] The first and second construction sequences are used for the motion vector difference (MMVD) merging mode.

[0207] 26. The method described under Clause 24 further includes:

[0208] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the FIFO table, a pairwise average candidate, or a zero motion vector.

[0209] 27. The method according to Clause 26, wherein,

[0210] The motion information of the previously encoded blocks is stored in a FIFO table and used as the motion vector predictor for the current encoding unit.

[0211] 28. The method according to any one of clauses 26 and 27, wherein

[0212] Motion information associated with coding units that are not sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP candidate.

[0213] 29. The method according to Clause 26, wherein,

[0214] The pairwise averaged candidate is generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidate is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0215] 30. The method according to Clause 26, wherein,

[0216] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0217] 31. A video processing apparatus, comprising:

[0218] A memory that stores a set of instructions; and

[0219] One or more processors, the one or more processors being configured to execute a set of instructions to cause the device to perform:

[0220] Insert a set of spatial merging candidates into the merging candidate list of the coded blocks.

[0221] Among them, the space merging candidates are inserted in the following order: top adjacent block, left adjacent block, top adjacent block, left adjacent block and top left adjacent block.

[0222] 32. The apparatus according to clause 31, wherein the one or more processors are configured to execute the instruction set to cause the apparatus to further perform:

[0223] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

[0224] 33. The apparatus according to clause 32, wherein,

[0225] Motion information of the previously encoded blocks is stored in a FIFO table and used as a motion vector predictor for the current encoding unit.

[0226] 34. The equipment according to any one of clauses 32 and 33, wherein

[0227] The motion information candidate associated with the coding unit that is not a sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP.

[0228] 35. The equipment as described in Clause 32, wherein,

[0229] The pairwise averaged candidates are generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidates are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0230] 36. The device as described in Clause 32, wherein,

[0231] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0232] 37. A video processing apparatus, comprising:

[0233] A memory that stores a set of instructions; and

[0234] One or more processors, the one or more processors being configured to execute a set of instructions to cause the device to perform:

[0235] Based on the preset limit on the number of spatial merging candidates, a set of spatial merging candidates are inserted into the merging candidate list of the coding block.

[0236] If the quantity limit is 2, then the space merging candidates in this group are inserted into the merge candidate list in the following order: the upper adjacent block, the left adjacent block; and

[0237] If the number is limited to 3, the space merging candidates will be inserted into the merging candidate list in the following order: top adjacent block, left adjacent block, top adjacent block.

[0238] 38. The device as described in Clause 37, wherein

[0239] In response to encoding the current image using past and future reference images according to the display order, the number of spatial merging candidates is set to a first value; and

[0240] In response to the fact that the current image has been encoded according to the display order and using past reference images, the number of the space merging candidates is set to a second value that is less than the first value.

[0241] 39. The apparatus according to clause 37, wherein the one or more processors are configured to execute the instruction set to cause the apparatus to further perform:

[0242] Mark the number of space merge candidates inserted into the merge candidate list.

[0243] 40. The apparatus according to any one of clauses 37-39, wherein the one or more processors are configured to execute the instruction set to cause the apparatus to further perform:

[0244] Add at least one of the following to the merge candidate list: a temporal merge candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the FIFO table, a pairwise average candidate, or a zero motion vector.

[0245] 41. The device as described in Clause 40, wherein

[0246] The motion information of the previously encoded blocks is stored in a FIFO table and used as the motion vector predictor for the current encoding unit.

[0247] 42. The equipment according to any one of clauses 40 and 41, wherein,

[0248] Motion information associated with coding units that are not sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP candidate.

[0249] 43. The apparatus according to clause 40, wherein,

[0250] The pairwise averaged candidates are generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidates are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0251] 44. The device according to clause 40, wherein,

[0252] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0253] 45. A video processing apparatus, comprising:

[0254] A memory that stores a set of instructions; and

[0255] One or more processors, the one or more processors being configured to execute a set of instructions to cause the device to perform:

[0256] A set of spatial merging candidates is inserted into the merging candidate list of the coded blocks, where:

[0257] When the first encoding mode is applied to the encoded block, the space merging candidates are inserted according to the first construction order; and

[0258] When the second encoding mode is applied to the encoded block, the space merging candidates are inserted according to the second construction order.

[0259] The first build order is different from the second build order.

[0260] 46. ​​The apparatus according to clause 45, wherein the first encoding mode and the second encoding mode are two different modes selected from the following:

[0261] The modes include standard merging mode, motion vector difference merging mode (MMVD), and triangular partitioning mode (TPM).

[0262] 47. The device as described in Clause 46, wherein,

[0263] In MMVD, a candidate for merging is first selected from the list of candidates for merging, then refined using the marked motion vector difference (MVD) information, and a merging candidate flag is marked to specify which of the two MMVD candidates to use as the base motion vector.

[0264] 48. The equipment according to any one of clauses 45 and 46, wherein,

[0265] In TPM, the coding unit is uniformly divided into two triangular partitions using at least one of diagonal partitioning or anti-diagonal partitioning.

[0266] 49. The device as described in Clause 46, wherein

[0267] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor from the FIFO table, a pairwise average candidate, or a zero motion vector.

[0268] 50. The device as described in Clause 49, wherein,

[0269] The motion information of the previously encoded blocks is stored in a FIFO table and used as the motion vector predictor for the current encoding unit.

[0270] 51. The equipment according to any one of clauses 49 and 50, wherein,

[0271] Motion information associated with coding units that are not sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP candidate.

[0272] 52. The device as described in Clause 49, wherein,

[0273] The pairwise averaged candidates are generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidates are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0274] 53. The equipment as described in Clause 49, wherein,

[0275] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0276] 54. A video processing apparatus, comprising:

[0277] A memory that stores a set of instructions; and

[0278] One or more processors, the one or more processors being configured to execute a set of instructions to cause the device to perform:

[0279] A set of spatial merging candidates is inserted into the merging candidate list of the coded blocks, where:

[0280] When the encoded block is part of a low-latency image, insert the group of spatial merge candidates according to the first build order; and

[0281] When the encoded block is part of a non-low-latency image, the group of spatial merging candidates is inserted according to the second build order.

[0282] The first build order is different from the second build order.

[0283] 55. The device as described in Clause 54, wherein,

[0284] The first and second construction sequences are used for the motion vector difference (MMVD) merging mode.

[0285] 56. The apparatus according to clause 54, wherein the one or more processors are configured to execute the instruction set to cause the apparatus to further perform:

[0286] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the FIFO table, a pairwise average candidate, or a zero motion vector.

[0287] 57. The device as described in Clause 56, wherein,

[0288] The motion information of the previously encoded blocks is stored in a FIFO table and used as the motion vector predictor for the current encoding unit.

[0289] 58. The equipment according to any one of clauses 56 and 57, wherein,

[0290] Motion information associated with coding units that are not sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP candidate.

[0291] 59. The equipment as described in Clause 56, wherein,

[0292] The pairwise averaged candidate is generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidate is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0293] 60. The device as described in Clause 56, wherein,

[0294] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0295] 61. A non-transitory computer-readable medium storing an instruction set executable by at least one processor of a computer to cause the computer to perform a video processing method, the method comprising:

[0296] Insert a set of spatial merging candidates into the merging candidate list of the coded blocks.

[0297] Among them, the space merging candidates are inserted in the following order: top adjacent block, left adjacent block, top adjacent block, left adjacent block and top left adjacent block.

[0298] 62. The non-transitory computer-readable medium according to clause 61, wherein the computer is executable with a set of instructions to cause the computer to further perform:

[0299] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the first-in-first-out (FIFO) table, a pairwise average candidate, or a zero motion vector.

[0300] 63. The non-transitory computer-readable medium as described in Clause 62, wherein,

[0301] Motion information of the previously encoded blocks is stored in a FIFO table and used as a motion vector predictor for the current encoding unit.

[0302] 64. The non-transitory computer-readable medium according to any one of clauses 62 and 63, wherein,

[0303] The motion information candidate associated with the coding unit that is not a sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP.

[0304] 65. The non-transitory computer-readable medium as described in Clause 62, wherein,

[0305] The pairwise averaged candidates are generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidates are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0306] 66. The non-transitory computer-readable medium as described in Clause 62, wherein,

[0307] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0308] 67. A non-transitory computer-readable medium storing an instruction set executable by at least one processor of a computer to cause the computer to perform a video processing method, the method comprising:

[0309] Based on the preset limit on the number of spatial merging candidates, a set of spatial merging candidates are inserted into the merging candidate list of the coding block.

[0310] If the quantity limit is 2, then the space merging candidates in this group are inserted into the merge candidate list in the following order: the upper adjacent block, the left adjacent block; and

[0311] If the number is limited to 3, the space merging candidates will be inserted into the merging candidate list in the following order: top adjacent block, left adjacent block, top adjacent block.

[0312] 68. The non-transitory computer-readable medium as described in Clause 67, wherein,

[0313] In response to encoding the current image using past and future reference images according to the display order, the number of spatial merging candidates is set to a first value; and

[0314] In response to the fact that the current image has been encoded according to the display order and using past reference images, the number of the space merging candidates is set to a second value that is less than the first value.

[0315] 69. The non-transitory computer-readable medium as described in Clause 67, wherein the computer is executable with a set of instructions to cause the computer to further perform:

[0316] Mark the number of space merge candidates inserted into the merge candidate list.

[0317] 70. A non-transitory computer-readable medium according to any one of clauses 67-69, wherein at least one processor is configured to execute the instruction set to cause the computer to further perform:

[0318] Add at least one of the following to the merge candidate list: a temporal merge candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the FIFO table, a pairwise average candidate, or a zero motion vector.

[0319] 71. The non-transitory computer-readable medium as described in Clause 70, wherein,

[0320] The motion information of the previously encoded blocks is stored in a FIFO table and used as the motion vector predictor for the current encoding unit.

[0321] 72. The non-transitory computer-readable medium according to any one of clauses 70 and 71, wherein

[0322] Motion information associated with coding units that are not sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP candidate.

[0323] 73. The non-transitory computer-readable medium as described in Clause 70, wherein,

[0324] The pairwise averaged candidates are generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidates are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0325] 74. The non-transitory computer-readable medium as described in Clause 70, wherein,

[0326] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0327] 75. A non-transitory computer-readable medium storing an instruction set executable by at least one processor of a computer to cause the computer to perform a video processing method, the method comprising:

[0328] A set of spatial merging candidates is inserted into the merging candidate list of the coded blocks, where:

[0329] When the first encoding mode is applied to the encoded block, the space merging candidates are inserted according to the first construction order; and

[0330] When the second encoding mode is applied to the encoded block, the space merging candidates are inserted according to the second construction order.

[0331] The first build order is different from the second build order.

[0332] 76. The non-transitory computer-readable medium as described in Clause 75, wherein the first encoding mode and the second encoding mode are two different modes selected from:

[0333] The modes include standard merging mode, motion vector difference merging mode (MMVD), and triangular partitioning mode (TPM).

[0334] 77. The non-transitory computer-readable medium as described in Clause 76, wherein,

[0335] In MMVD, a candidate for merging is first selected from the list of candidates for merging, then refined using the marked motion vector difference (MVD) information, and a merging candidate flag is marked to specify which of the two MMVD candidates to use as the base motion vector.

[0336] 78. The non-transitory computer-readable medium as described in Clause 76, wherein,

[0337] In TPM, the coding unit is uniformly divided into two triangular partitions using at least one of diagonal partitioning or anti-diagonal partitioning.

[0338] 79. The non-transitory computer-readable medium according to any one of clauses 75 and 76, wherein

[0339] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor from the FIFO table, a pairwise average candidate, or a zero motion vector.

[0340] 80. The non-transitory computer-readable medium as described in Clause 79, wherein,

[0341] The motion information of the previously encoded blocks is stored in a FIFO table and used as the motion vector predictor for the current encoding unit.

[0342] 81. The non-transitory computer-readable medium according to any one of clauses 79 and 80, wherein

[0343] Motion information associated with coding units that are not sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP candidate.

[0344] 82. The non-transitory computer-readable medium according to claim 79, wherein,

[0345] The pairwise averaged candidates are generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidates are added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0346] 83. The non-transitory computer-readable medium as described in Clause 79, wherein,

[0347] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0348] 84. A non-transitory computer-readable medium storing an instruction set executable by at least one processor of a computer to cause the computer to perform a video processing method, the method comprising:

[0349] A set of spatial merging candidates is inserted into the merging candidate list of the coded blocks, where:

[0350] When the encoded block is part of a low-latency image, insert the group of spatial merge candidates according to the first build order; and

[0351] When the encoded block is part of a non-low-latency image, the group of spatial merging candidates is inserted according to the second build order.

[0352] The first build order is different from the second build order.

[0353] 85. The non-transitory computer-readable medium as described in Clause 84, wherein,

[0354] The first and second construction sequences are used for the motion vector difference (MMVD) merging mode.

[0355] 86. The non-transitory computer-readable medium as described in Clause 84, wherein the computer is executable with a set of instructions to cause the computer to perform further actions:

[0356] Add at least one of the following to the merge candidate list: a time merge candidate from the co-located coding unit, a history-based motion vector predictor (HMVP) from the FIFO table, a pairwise average candidate, or a zero motion vector.

[0357] 87. The non-transitory computer-readable medium as described in Clause 86, wherein,

[0358] The motion information of the previously encoded blocks is stored in a FIFO table and used as the motion vector predictor for the current encoding unit.

[0359] 88. The non-transitory computer-readable medium pursuant to any one of clauses 86 and 87, wherein,

[0360] Motion information associated with coding units that are not sub-block inter-frame coded is added to the last item of the FIFO table as a new HMVP candidate.

[0361] 89. The non-transitory computer-readable medium as described in Clause 86, wherein,

[0362] The pairwise averaged candidate is generated by averaging the candidate pairs in the merge candidate list, and the pairwise averaged candidate is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

[0363] 90. The non-transitory computer-readable medium as described in Clause 86, wherein

[0364] Insert the zero motion vector at the end of the merge candidate list until the maximum number of merge candidates is reached.

[0365] In the foregoing specification, embodiments have been described with reference to numerous specific details, which may vary with different implementations. Certain adjustments and modifications can be made to the described embodiments. Other embodiments will be apparent to those skilled in the art, considering the specifications and practices of this application disclosed herein. The foregoing specification and embodiments are considered merely examples, and the true scope and spirit of this application are indicated by the claims. The sequence of steps shown in the figures is also intended for illustrative purposes only and is not intended to limit to any particular order of steps. Therefore, those skilled in the art will understand that these steps may be performed in different orders while implementing the same method.

[0366] Exemplary embodiments have been disclosed in the accompanying drawings and description. However, many variations and modifications can be made to these embodiments. Therefore, although specific terms have been used, they are used in a general and descriptive sense only and not for limiting purposes.

Claims

1. A method for encoding a video sequence into a bitstream, the method comprising: Receive video sequences; Encoding one or more images of the video sequence; and generating a bitstream, wherein the encoding includes: Insert a set of spatial merging candidates into the merging candidate list of the coded block, where: When the coded block is part of a low-latency image, this set of spatial merge candidates is inserted according to the first construction order; and When the encoded block is part of a non-low-latency image, the set of spatial merge candidates is inserted according to a second construction order, the first construction order being different from the second construction order.

2. The method according to claim 1, wherein The first and second construction sequences are used for merging modes with motion vector difference (MMVD).

3. The method according to claim 1, further comprising: Add at least one of the following to the merge candidate list: temporal merge candidate from the co-located coding unit, historical motion vector prediction (HMVP) from the first-in-first-out (FIFO) table, pairwise average candidate, and zero motion vector.

4. The method according to claim 3, wherein, When an HMVP is added from the FIFO table, motion information of the previously encoded block is stored in the FIFO table and used as the motion vector prediction value for the current encoding unit.

5. The method according to claim 3, wherein When an HMVP is added from the FIFO table, motion information associated with the coding unit that is not a sub-block inter-frame coding is added to the last entry of the FIFO table as a new HMVP candidate.

6. The method according to claim 3, wherein When adding the pairwise average candidate, the pairwise average candidate is generated by averaging multiple candidate pairs in the merge candidate list, and the pairwise average candidate is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

7. The method according to claim 3, wherein, When the zero motion vector is added, it is inserted at the end of the merge candidate list until the maximum number of merge candidates is reached.

8. A method for decoding a bitstream to output one or more images of a video stream, the method comprising: Receive bitstream; and Decoding one or more images using the encoding information of the bitstream, wherein decoding includes: Insert a set of spatial merging candidates into the merging candidate list of the coded block, where: When the coded block is part of a low-latency image, this set of spatial merge candidates is inserted according to the first construction order; and When the encoded block is part of a non-low-latency image, the set of spatial merge candidates is inserted according to a second construction order, the first construction order being different from the second construction order.

9. The method according to claim 8, wherein The first and second construction sequences are used for merging modes with motion vector difference (MMVD).

10. The method of claim 8, further comprising: Add at least one of the following to the merge candidate list: temporal merge candidate from the co-located coding unit, historical motion vector prediction (HMVP) from the first-in-first-out (FIFO) table, pairwise average candidate, and zero motion vector.

11. The method according to claim 10, wherein, When an HMVP is added from the FIFO table, motion information of the previously encoded block is stored in the FIFO table and used as the motion vector prediction value for the current encoding unit.

12. The method of claim 10, wherein When an HMVP is added from the FIFO table, motion information associated with the coding unit that is not a sub-block inter-frame coding is added to the last entry of the FIFO table as a new HMVP candidate.

13. The method of claim 10, wherein When adding the pairwise average candidate, the pairwise average candidate is generated by averaging multiple candidate pairs in the merge candidate list, and the pairwise average candidate is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

14. The method of claim 10, wherein When the zero motion vector is added, it is inserted at the end of the merge candidate list until the maximum number of merge candidates is reached.

15. A non-transitory computer-readable storage medium storing an instruction set and a bit stream, the instruction set being executable by one or more processors in a method to generate the bit stream, the method comprising: Insert a set of spatial merging candidates into the merging candidate list of the coded block, where: When the coded block is part of a low-latency image, this set of spatial merge candidates is inserted according to the first construction order; and When the encoded block is part of a non-low-latency image, the set of spatial merge candidates is inserted according to a second construction order, the first construction order being different from the second construction order.

16. The non-transitory computer-readable storage medium according to claim 15, wherein The first and second construction sequences are used for merging modes with motion vector difference (MMVD).

17. The non-transitory computer-readable storage medium according to claim 15, wherein, The method further includes: Add at least one of the following to the merge candidate list: temporal merge candidate from the co-located coding unit, historical motion vector prediction (HMVP) from the first-in-first-out (FIFO) table, pairwise average candidate, and zero motion vector.

18. The non-transitory computer-readable storage medium according to claim 17, wherein, When an HMVP is added from the FIFO table, motion information of the previously encoded block is stored in the FIFO table and used as the motion vector prediction value for the current encoding unit.

19. The non-transitory computer-readable storage medium according to claim 17, wherein When an HMVP is added from the FIFO table, motion information associated with the coding unit that is not a sub-block inter-frame coding is added to the last entry of the FIFO table as a new HMVP candidate.

20. The non-transitory computer-readable storage medium of claim 17, wherein when the pairwise average candidate is added, the pairwise average candidate is generated by averaging multiple candidate pairs in the merge candidate list, and the pairwise average candidate is added to the merge candidate list after one or more HMVPs are added to the merge candidate list in response to the merge candidate list being incomplete.

Images