Method and apparatus for candidate derivation of affine merge mode in video coding

Abstract

Methods, apparatuses, and non-transitory storage media for video decoding and encoding are provided. In a decoding method, a decoder obtains a normalized position for an affine coding unit (CU) at a non-adjacent position based on a storage granularity, where the non-adjacent position is obtained by scanning a region that is not adjacent to the current CU. Further, the decoder determines whether effective affine model information is stored for the normalized location, where the effective affine model information includes the location of the affine CU, and in response to determining that the effective affine model information is stored for the normalized location, the decoder determines whether the effective affine model information includes the location of the affine CU. Affine model information for the affine CU is derived based on the effective affine model information stored for the normalized location.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims priority from U.S. Provisional Application No. 63 / 358,511, filed on July 5, 2022, entitled “Methods and Devices for Candidate Derivation for Affine Merge Mode in Video Coding,” the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] The present disclosure relates to video coding and compression, and more particularly, but not limited to, to methods and apparatus for improving affine merging candidate derivation for affine motion prediction mode in video encoding or decoding processes. Background Art

[0004] Various video codec technologies can be used to compress video data. Video coding is performed according to one or more video codec standards. For example, today, some well-known video codec standards include Universal Video Codec (VVC), High Efficiency Video Codec (HEVC, also known as H.265 or MPEG-H Part 2) and Advanced Video Codec (AVC, also known as H.264 or MPEG-4 Part 10) jointly developed by ISO / IEC MPEG and ITU-TVECG. AOMedia Video 1 (AV1) was developed by the Alliance for Open Media (AOM) as a successor to its previous standard VP9. Audio Video Coding (AVS) refers to digital audio and digital video compression standards, another video compression standard series developed by the Chinese Audio and Video Coding Standard Working Group. Most existing video codec standards are based on the well-known hybrid video codec framework, that is, block-based prediction methods (e.g., inter-frame prediction, intra-frame prediction) are used to reduce the redundancy present in video images or sequences, and transform coding is used to compress the energy of prediction errors. An important goal of video coding technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing video quality degradation.

[0005] The first generation of AVS standards includes the Chinese national standard "Information Technology Advanced Audio and Video Codec Part 2: Video" (referred to as AVS1) and "Information Technology Advanced Audio and Video Codec Part 16: Broadcasting and Television Video" (referred to as AVS+). Compared with the MPEG-2 standard, it can provide about 50% bit rate savings at the same perceptual quality. The second generation of AVS standards includes the Chinese national standard "Information Technology High Efficiency Multimedia Codec" (referred to as AVS2), which is mainly aimed at the transmission of ultra-high-definition television programs. The codec efficiency of AVS2 is twice that of AVS+. In May 2016, AVS2 was released as a Chinese national standard. At the same time, the video part of the AVS2 standard was submitted by the Institute of Electrical and Electronics Engineers (IEEE) as an international standard for application. The AVS3 standard is a new generation of video codec standards for UHD video applications, aiming to exceed the codec efficiency of the latest international standard HEVC. In March 2019, at the 68th AVS meeting, the AVS3-P2 baseline was completed, which provides about 30% bit rate savings compared to the HEVC standard. Currently, there is a reference software called High Performance Model (HPM) maintained by the AVS group to demonstrate a reference implementation of the AVS3 standard. Summary of the invention

[0006] This disclosure provides examples of techniques related to improving motion vector candidate derivation for motion prediction modes in a video encoding or decoding process.

[0007] According to a first aspect of the present disclosure, a method for video decoding is provided. In the method for video decoding, a decoder may obtain a normalized position for an affine coding unit (CU) at a non-adjacent position based on a storage granularity, wherein the non-adjacent position is obtained by scanning an area that is not adjacent to a current CU. In addition, the decoder may obtain a determination as to whether valid affine model information is stored for the normalized position, wherein the valid affine model information may include the position of the affine CU. In addition, the decoder may derive the affine model information for the affine CU based on the valid affine model information stored for the normalized position in response to determining that the valid affine model information is stored for the normalized position.

[0008] According to a second aspect of the present disclosure, a method for video encoding is provided. In the method for video encoding, the encoder may store a normalized position of an affine coding unit (CU) at a non-adjacent position based on a storage granularity, wherein the non-adjacent position is obtained by scanning an area that is not adjacent to the current CU. In addition, the encoder may determine whether valid affine model information is stored for the normalized position, wherein the valid affine model information may include the position of the affine CU. In addition, the encoder may derive the affine model information for the affine CU based on the valid affine model information stored for the normalized position in response to determining that the valid affine model information is stored for the normalized position.

[0009] According to a third aspect of the present disclosure, a method for video decoding is provided. In video decoding, the decoder may scan the neighboring area of ​​the current CU to obtain a previously encoded neighboring affine CU, wherein the neighboring affine CU includes a plurality of internal sub-blocks. In addition, the decoder may derive a plurality of first affine model parameters based on the neighboring affine CU. In addition, the decoder may derive a plurality of second affine model parameters based on a second regression model, wherein the input of the second linear regression model may include motion information of a second internal sub-block set and motion information of a plurality of neighboring sub-blocks of the current CU, wherein the second internal sub-block set is selected from the plurality of internal sub-blocks, and the number of the second internal sub-block set is less than or equal to the number of all the plurality of internal sub-blocks included in the neighboring affine CU, and wherein the plurality of neighboring sub-blocks are located in a neighboring area adjacent to the current CU. In addition, the decoder may obtain a plurality of CPMVs based on the plurality of first affine model parameters and the plurality of second affine model parameters.

[0010] According to a fourth aspect of the present disclosure, a method for video encoding is provided. In the method for video encoding, the encoder may scan the neighboring area of ​​the current CU to obtain a previously encoded neighboring affine CU, wherein the neighboring affine CU includes a plurality of internal sub-blocks. In addition, the encoder may derive a plurality of first affine model parameters based on the neighboring affine CU, and derive a plurality of second affine model parameters based on a second regression model, wherein the input of the second linear regression model includes motion information of a second internal sub-block set and motion information of a plurality of neighboring sub-blocks of the current CU, wherein the second internal sub-block set is selected from the plurality of internal sub-blocks, and the number of the second internal sub-block set is less than or equal to the number of all the plurality of internal sub-blocks included in the neighboring affine CU, and wherein the plurality of neighboring sub-blocks are located in a neighboring area adjacent to the current CU. In addition, the encoder may obtain a plurality of CPMVs based on the plurality of first affine model parameters and the plurality of second affine model parameters.

[0011] According to a fifth aspect of the present disclosure, a device for video decoding is provided. The device includes one or more processors and a memory, the memory is coupled to the one or more processors and is configured to store instructions executable by the one or more processors. In addition, when executing the instructions, the one or more processors are configured to perform the method according to the first aspect or the third aspect above.

[0012] According to a sixth aspect of the present disclosure, a device for video encoding is provided. The device includes one or more processors and a memory, the memory is coupled to the one or more processors and is configured to store instructions executable by the one or more processors. In addition, the one or more processors are configured to execute the method according to the second aspect or the fourth aspect when executing the instructions.

[0013] According to the seventh aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer-executable instructions is provided, wherein the computer-executable instructions, when executed by one or more computer processors, cause the one or more computer processors to receive a bit stream and execute the method according to the first aspect or the third aspect above.

[0014] According to the eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer-executable instructions is provided, and when the computer-executable instructions are executed by one or more computer processors, the one or more computer processors execute the method according to the second aspect or the fourth aspect above to encode the current CU into a bit stream and send the bit stream.

[0015] According to a ninth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided for storing a bit stream to be decoded by the method of any one of the first aspect or the third aspect.

[0016] According to a tenth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which stores a bit stream generated by the method of any one of the second aspect or the fourth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] A more particular description of examples of the present disclosure will be presented by reference to specific examples shown in the accompanying drawings. Given that these drawings depict only some examples and are therefore not to be considered limiting in scope, the examples will be described and explained with additional specificity and detail through use of the accompanying drawings.

[0018] Figure 1A is a block diagram illustrating a system for encoding and decoding video blocks according to some examples of the present disclosure.

[0019] Figure 1Bis a block diagram of an encoder according to some examples of the present disclosure.

[0020] Figure 1C-1F is a block diagram showing how to recursively divide a frame into multiple video blocks of different sizes and shapes according to some examples of the present disclosure.

[0021] Figure 1G is a block diagram illustrating an exemplary video encoder according to some examples of the present disclosure.

[0022] Figure 2A is a block diagram of a decoder according to some examples of the present disclosure.

[0023] Figure 2B is a block diagram illustrating an exemplary video decoder according to some examples of the present disclosure.

[0024] Figure 3A is a diagram illustrating block partitioning in a multi-type tree structure according to some examples of the present disclosure.

[0025] Figure 3B is a diagram illustrating block partitioning in a multi-type tree structure according to some examples of the present disclosure.

[0026] Figure 3C is a diagram illustrating block partitioning in a multi-type tree structure according to some examples of the present disclosure.

[0027] Figure 3D is a diagram illustrating block partitioning in a multi-type tree structure according to some examples of the present disclosure.

[0028] Figure 3E A diagram illustrating block partitioning in a multi-type tree structure according to some examples of the present disclosure.

[0029] Figure 4A A 4-parameter affine model according to some examples of the present disclosure is shown.

[0030] Figure 4B A 4-parameter affine model according to some examples of the present disclosure is shown.

[0031] Figure 4C The positions of spatial merging candidates according to some examples of the present disclosure are shown.

[0032] Figure 4D Candidate pairs considered for redundancy checking of spatial merging candidates according to some examples of the present disclosure are shown.

[0033] Figure 4E Motion vector scaling for temporal merging candidates according to some examples of the present disclosure is shown.

[0034] Figure 4FThe temporal merge candidate C according to some examples of the present disclosure is shown. 0 and C 1 candidate locations.

[0035] Figure 5 A 6-parameter affine model according to some examples of the present disclosure is shown.

[0036] Figure 6 Neighboring neighboring blocks for inherited affine merging candidates according to some examples of the present disclosure are shown.

[0037] Figure 7 Neighboring neighboring blocks used to construct affine merge candidates according to some examples of the present disclosure are shown.

[0038] Figure 8 Non-adjacent neighboring blocks for inheriting affine merging candidates according to some examples of the present disclosure are shown.

[0039] Fig. 9 The derivation of constructed affine merging candidates using non-adjacent neighboring blocks according to some examples of the present disclosure is shown.

[0040] Fig.10 Vertical scanning of non-adjacent neighboring blocks according to some examples of the present disclosure is shown.

[0041] Fig.11 Parallel scanning of non-adjacent neighboring blocks according to some examples of the present disclosure is shown.

[0042] Fig.12 Combined vertical and parallel scanning of non-adjacent neighboring blocks is shown according to some examples of the present disclosure.

[0043] Fig.13A Neighboring blocks having the same size as a current block according to some examples of the present disclosure are shown.

[0044] Fig. 13B Neighboring blocks having different sizes from the current block according to some examples of the present disclosure are shown.

[0045] Fig.14A An example in which a lower left block or an upper right block of a bottommost block or a rightmost block of a previous distance according to some examples of the present disclosure is used as a bottommost block or a rightmost block of a current distance is shown.

[0046] Fig. 14B An example in which a left block or a top block of a bottommost block or a rightmost block of a previous distance according to some examples of the present disclosure is used as a bottommost block or a rightmost block of a current distance is shown.

[0047] Fig.15AScan positions at lower left and upper right positions for an upper non-adjacent neighboring block and a left non-adjacent neighboring block according to some examples of the present disclosure are shown.

[0048] Fig. 15B The scanning position at the lower right position for both the above non-adjacent neighboring block and the left non-adjacent neighboring block according to some examples of the present disclosure is shown.

[0049] Fig. 15C The scanning position at the lower left position for both the above non-adjacent neighboring block and the left non-adjacent neighboring block according to some examples of the present disclosure is shown.

[0050] Fig.15D The scanning position at the upper right position for both the above non-adjacent neighboring block and the left non-adjacent neighboring block according to some examples of the present disclosure is shown.

[0051] Fig.16 A simplified scanning process for deriving build merge candidates according to some examples of the present disclosure is shown.

[0052] Fig.17A Spatial neighbors for deriving inherited affine merge candidates according to some examples of the present disclosure are shown.

[0053] Fig. 17B Spatial neighbors for deriving and constructing affine merge candidates according to some examples of the present disclosure are shown.

[0054] Fig.18 An example of an inheritance-based derivation method for deriving affine construction candidates according to some examples of the present disclosure is shown.

[0055] Fig.19 Templates according to some examples of the present disclosure and reference samples of the templates in reference list 0 and reference list 1 are shown.

[0056] Fig. 20 Templates according to some examples of the present disclosure and reference samples of the template for a block having sub-block motion using motion information of a sub-block of a current block are shown.

[0057] Fig.21 An example is shown in which the non-adjacent spatial region is restricted to a region of half the CTU size above and half the CTU size to the left of the current CTU according to some examples of the present disclosure.

[0058] Fig.22A A storage method of directly saving affine motion information about an affine coding block CTU according to some examples of the present disclosure is shown.

[0059] Fig. 22BA storage method of projecting and saving affine motion information at each sub-block according to some examples of the present disclosure is shown.

[0060] Fig.23 An example of deriving regular / translational motion at each 4×4 regular block using a center point according to some examples of the present disclosure is shown.

[0061] Fig.24 is a diagram illustrating a computing environment coupled to a user interface according to some examples of the present disclosure.

[0062] Fig.25 An example of storing motion information of an affine encoded block at a granularity greater than a minimum affine block size according to some examples of the present disclosure is shown.

[0063] Fig.26 Merge mode (MMVD) search points with MVD in L0 reference and L1 reference respectively are shown according to some examples of the present disclosure.

[0064] Fig.27A An example of motion storage for non-adjacent spatial neighbors including affine neighboring CUs and non-affine neighboring CUs when an allowable non-adjacent spatial region is outside a current coding tree unit (CTU) according to some examples of the present disclosure is shown.

[0065] Fig.27B Examples of motion storage for non-adjacent spatial neighbors including affine neighboring CU and non-affine neighboring CU motion storage in a line buffer according to some examples of the present disclosure are shown.

[0066] Fig.28 Examples of projected or cropped non-adjacent neighbor positions when the scanned non-adjacent neighbor positions are outside the allowable spatial region according to some examples of the present disclosure are shown.

[0067] Fig.29 Another example of projected / cropped non-adjacent neighbor positions is shown in accordance with some examples of the present disclosure when the scanned non-adjacent neighbor positions are outside the allowed spatial region (e.g., outside the current CTU and available line buffers).

[0068] Figure 30A-Figure 30C It is shown that non-adjacent affine motion information is obtained from existing stored translation motion information according to some examples of the present disclosure.

[0069] Figure 31A-Figure 31C Some examples according to the present disclosure are shown by using Figure 30A-Figure 30C Less storage information is required to obtain non-adjacent affine motion information from the existing stored translation motion information.

[0070] Fig.32Neighboring 4×4 sub-blocks are shown for regression-based motion vector field (RMVF) parameter derivation according to some examples of the present disclosure.

[0071] Fig.33 is a flowchart illustrating a method for video decoding according to some examples of the present disclosure.

[0072] Fig.34 is a diagram showing some examples according to the present disclosure. Fig.33 The method for video decoding shown in FIG. 1 corresponds to a flow chart of a method for video encoding.

[0073] Fig.35 is a flowchart illustrating a method for video decoding according to some examples of the present disclosure.

[0074] Fig.36 is a diagram showing some examples according to the present disclosure. Fig.35 The method for video decoding shown in FIG. 1 corresponds to a flow chart of a method for video encoding. DETAILED DESCRIPTION

[0075] Reference will now be made in detail to specific embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0076] The terms used in this disclosure are only suitable for the purpose of describing specific embodiments and are not intended to limit the present disclosure. In the present disclosure and the appended claims, singular forms ("a / an", "the", "said", etc.) are also intended to include plural forms, unless other meanings are clearly indicated throughout the disclosure. It should also be understood that the term "and / or" used in this disclosure indicates and includes one or any or all possible combinations of the listed multiple related items.

[0077] References throughout this specification to "one embodiment," "an embodiment," "an example," "some embodiments," "some examples," or similar language mean that the particular feature, structure, or characteristic being described is included in at least one embodiment or example. Unless explicitly stated otherwise, features, structures, elements, or characteristics described in conjunction with one or some embodiments are also applicable to other embodiments.

[0078] Throughout the disclosure, unless otherwise expressly stated, the terms "first", "second", "third", etc. are used merely as references to related elements (e.g., devices, components, compositions, steps, etc.), without implying any spatial or temporal order. For example, "first device" and "second device" may refer to two separately formed devices, or two parts, components, or operating states of the same device, and may be named arbitrarily.

[0079] The terms "module", "sub-module", "circuit", "sub-circuit", "circuitry", "sub-circuitry", "unit", or "sub-unit" may include a memory (shared, dedicated, or group) storing code or instructions that may be executed by one or more processors. A module may include one or more circuits with or without stored code or instructions. A module or circuit may include one or more components connected directly or indirectly. These components may or may not be physically attached to each other or located adjacent to each other.

[0080] As used herein, the terms "if" or "when" may be understood to mean "based on" or "in response to" depending on the context. These terms, if they appear in a claim, may not indicate that the associated limitation or feature is conditional or optional. For example, a method may include the following steps: i) when or if condition X exists, perform function or action X', and ii) when or if condition Y exists, perform function or action Y'. The method may be implemented with the ability to perform function or action X' and the ability to perform function or action Y'. Therefore, both functions X' and Y' may be performed at different times during multiple executions of the method.

[0081] A unit or module may be implemented purely by software, purely by hardware, or by a combination of hardware and software. In a pure software implementation, for example, a unit or module may include functionally related code blocks or software components that are linked together directly or indirectly to perform a specific function.

[0082] Figure 1A is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1A As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0083] In some embodiments, the target device 14 may receive the encoded video data to be decoded via the link 16. The link 16 may include any type of communication medium or device capable of moving the encoded video data from the source device 12 to the target device 14. In one example, the link 16 may include a communication medium that enables the source device 12 to send the encoded video data directly to the target device 14 in real time. The encoded video data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to the target device 14. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network (e.g., a local area network, a wide area network, or a global network such as the Internet). The communication medium may include a router, a switch, a base station, or any other device that may be useful in facilitating communication from the source device 12 to the target device 14.

[0084] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. Subsequently, the encoded video data in the storage device 32 may be accessed by the target device 14 via the input interface 28. The storage device 32 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a digital versatile disk (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device 32 may correspond to a file server or another intermediate storage device that can store the encoded video data generated by the source device 12. The target device 14 can access the stored video data from the storage device 32 via streaming or downloading. The file server can be any type of computer capable of storing encoded video data and sending the encoded video data to the target device 14. Exemplary file servers include a web server (e.g., for a website), a File Transfer Protocol (FTP) server, a Network Attached Storage (NAS) device, or a local disk drive. The target device 14 may access the encoded video data through any standard data connection suitable for accessing the encoded video data stored on the file server, including a wireless channel (e.g., a Wireless Fidelity (Wi-Fi) connection), a wired connection (e.g., a Digital Subscriber Line (DSL), a cable modem, etc.), or a combination of both. The transmission of the encoded video data from the storage device 32 may be a streaming transmission, a download transmission, or a combination of both a streaming transmission and a download transmission.

[0085] like Figure 1A As shown in , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include a source such as a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video, or a combination of such sources. As an example, if video source 18 is a camera of a security monitoring system, source device 12 and target device 14 may form a camera phone or a video phone. However, the embodiments described in this application may be generally applicable to video encoding and decoding, and may be applied to wireless and / or wired applications.

[0086] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback. Output interface 22 may also include a modem and / or a transmitter.

[0087] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or modem and receives encoded video data via link 16. The encoded video data communicated via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0088] In some embodiments, the target device 14 may include a display device 34, which may be an integrated display device and an external display device configured to communicate with the target device 14. The display device 34 displays the decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0089] The video encoder 20 and the video decoder 30 may operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, MPEG-4, Part 10, AVC), or an extension of such a standard. It should be understood that the present application is not limited to a particular video encoding / decoding standard and may be applicable to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 may be configured to encode video data according to any of these current or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 may be configured to decode video data according to any of these current or future standards.

[0090] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, respectively, such as one or more microprocessors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), discrete logic, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-volatile computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0091] Like HEVC, VVC is built on a block-based hybrid video codec framework. Figure 1B 1 is a block diagram illustrating a block-based video encoder according to some embodiments of the present disclosure. In the encoder 100, the input video signal is processed block by block (referred to as a coding unit (CU)). The encoder 100 may be as follows: Figure 1A The video encoder 20 shown in . In VTM-1.0, a CU can be up to 128×128 pixels. However, unlike HEVC, which partitions blocks based only on a quadtree, in VVC, a coding tree unit (CTU) is divided into multiple CUs to adapt to the changing local characteristics based on quadtrees / binary trees / ternary trees. In addition, the concept of multiple partition unit types in HEVC is removed, that is, there is no longer a distinction between CUs, prediction units (PUs), and transform units (TUs) in VVC; instead, each CU is always used as a basic unit for prediction and transformation without further division. In a multi-type tree structure, a CTU is first divided by a quadtree structure. Then, each quadtree leaf node can be further divided by a binary tree structure and a ternary tree structure.

[0092] Figure 3A-3E is a schematic diagram illustrating a multi-type tree partitioning mode according to some embodiments of the present disclosure. Figure 3A-3E Five types of segmentation are shown, including four-element segmentation ( Figure 3A ), vertical binary partition ( Figure 3B ), horizontal binary partition ( Figure 3C ), vertical expansion of the three-way partition ( Figure 3D ) and horizontally extended ternary partitioning ( Figure 3E ).

[0093] For each given video block, spatial prediction and / or temporal prediction may be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples of coded neighboring blocks in the same video picture / slice (which are called reference samples) to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in video signals. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses reconstructed pixels from coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index is additionally sent, which is used to identify which reference picture in the reference picture repository the temporal prediction signal comes from.

[0094] After spatial prediction and / or temporal prediction, the intra / inter mode decision circuit 121 in the encoder 100 selects the best prediction mode, for example, based on a rate-distortion optimization method. The block prediction factor 120 is then subtracted from the current video block; and the resulting prediction residual is decorrelated using the transform circuit 102 and the quantization circuit 104. The resulting quantized residual coefficients are dequantized by the dequantization circuit 116 and inversely transformed by the inverse transform circuit 118 to form a reconstructed residual, which is then added back to the predicted block to form a reconstructed signal for the CU. In addition, before the reconstructed CU is placed in the reference picture storage of the picture buffer 117 and used to encode and decode future video blocks, loop filtering 115, such as a deblocking filter, sample adaptive offset (SAO) and / or adaptive loop filter (ALF), may be applied to the reconstructed CU. In order to form an output video bitstream 114, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit 106 for further compression and packaging to form a bitstream.

[0095] For example, deblocking filters are available in the current versions of AVC, HEVC, and VVC. In HEVC, an additional loop filter called SAO is defined to further improve the coding efficiency. In the current version of the VVC standard, another loop filter called ALF is being actively studied and is likely to be included in the final standard.

[0096] These loop filter operations are optional. Performing these operations helps improve codec efficiency and visual quality. They can also be turned off as a decision presented by the encoder 100 to save computational complexity.

[0097] It should be noted that intra prediction is usually based on unfiltered reconstructed pixels, while inter prediction is based on filtered reconstructed pixels if the encoder 100 turns on these filter options.

[0098] Figure 2A is a block diagram illustrating a block-based video decoder 200 that can be used in conjunction with many video codec standards. The decoder 200 is similar to the one residing in Figure 1B The block-based video decoder 200 may be as follows: Figure 1A . In the decoder 200, the input video bitstream 201 is first decoded by entropy decoding 202 to derive quantization coefficient levels and prediction related information. The quantization coefficient levels are then processed by inverse quantization 204 and inverse transform 206 to obtain reconstructed prediction residuals. The block predictor mechanism implemented in the intra / inter mode selector 212 is configured to perform intra prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by summing the reconstructed prediction residuals from the inverse transform 206 and the prediction output generated by the block predictor mechanism using an adder 214.

[0099] The reconstructed block may further pass through a loop filter 209 before it is stored in a picture buffer 213 used as a reference picture repository. The reconstructed video in the picture buffer 213 may be sent to drive a display device and used to predict future video blocks. With the loop filter 209 turned on, a filtering operation is performed on these reconstructed pixels to derive a final reconstructed video output 222.

[0100] Figure 1G is a block diagram illustrating another exemplary video encoder 20 according to some embodiments described in the present application. The video encoder 20 may perform intra-frame prediction encoding and inter-frame prediction encoding of video blocks within a video frame. Intra-frame prediction encoding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction encoding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that the term "frame" may be used as a synonym for the term "image" or "picture" in the field of video coding and decoding.

[0101] like Figure 1GAs shown in FIG. 1 , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 also includes a motion estimation unit 42, a motion compensation unit 44, a partition unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 (such as a deblocking filter) may be located between the adder 62 and the DPB 64 to filter the block boundaries to remove blocking artifacts from the reconstructed video. In addition to using a deblocking filter, additional loop filters, such as a Sample Adaptive Offset (SAO) filter and / or an Adaptive in-Loop Filter (ALF), may be used to filter the output of adder 62. In some examples, the loop filter may be omitted, and the decoded video block may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed in one or more of the illustrated fixed or programmable hardware units.

[0102] Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from Figure 1A The video data memory 40 and the DPB 64 are obtained from the video source 18 shown. The DPB 64 is a buffer that stores reference video data (reference frames or pictures) for use by the video encoder 20 (e.g., in intra-frame or inter-frame prediction coding mode) when encoding video data. The video data memory 40 and the DPB 64 may be formed by any of a variety of memory devices. In various examples, the video data memory 40 may be on-chip with other components of the video encoder 20, or off-chip relative to those components.

[0103] like Figure 1GAs shown in , after receiving the video data, the partitioning unit 45 within the prediction processing unit 41 partitions the video data into video blocks. This partitioning operation may also include partitioning the video frame into strips, tiles (e.g., a set of video blocks) or other larger coding units (Coding Unit, CU) according to a predefined splitting structure associated with the video data (e.g., a quadtree (Quad-Tree, QT) structure). A video frame is or can be considered to be a two-dimensional array or matrix of samples with sample values. The samples in the array may also be referred to as pixels or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the video frame. The video frame may be divided into multiple video blocks by (for example) using QT partitioning. A video block is also or can be considered to be a two-dimensional array or matrix of samples with sample values, but the size of the video block is smaller than the video frame. The number of samples in the horizontal and vertical directions (or axes) of the video block defines the size of the video block. The video block may be further divided into one or more block partitions or sub-blocks (which may again form blocks) by, for example, iteratively using QT partitioning, binary-tree (BT) partitioning or ternary-tree (TT) partitioning or any combination thereof. It should be noted that the term "block" or "video block" as used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU) or a transform unit (TU) and / or may be or correspond to a corresponding block, such as a coding tree block (CTB), a coding block (CB), a prediction block (PB) or a transform block (TB) and / or correspond to a sub-block.

[0104] The prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., code rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block (e.g., prediction block) or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (such as motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

[0105] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block relative to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, e.g., to select an appropriate coding mode for each block of video data.

[0106] In some embodiments, motion estimation unit 42 determines the inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion for a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame, the prediction block in the reference frame relative to the current block being encoded in the current frame. The predetermined pattern may designate video frames in a sequence as P frames or B frames. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vectors determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine the block vector.

[0107] In terms of pixel differences, the prediction block of the video block may be or may correspond to a block or reference block of a reference frame that closely matches the video block to be encoded, and the pixel difference may be determined by the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. In some embodiments, the video encoder 20 may calculate values ​​for sub-integer pixel positions of the reference frame stored in the DPB 64. For example, the video encoder 20 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Thus, the motion estimation unit 42 may perform a motion search relative to the full pixel positions and the fractional pixel positions and output a motion vector with fractional pixel precision.

[0108] Motion estimation unit 42 calculates a motion vector for a video block in an inter-prediction coded frame by comparing the position of the video block to the position of a prediction block of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy encoding unit 56.

[0109] The motion compensation performed by the motion compensation unit 44 may involve extracting or generating a prediction block based on the motion vector determined by the motion estimation unit 42. After receiving the motion vector for the current video block, the motion compensation unit 44 may locate the prediction block pointed to by the motion vector in one of the reference frame lists, retrieve the prediction block from the DPB 64, and forward the prediction block to the adder 50. The adder 50 then forms a residual video block of pixel difference values ​​by subtracting the pixel values ​​of the prediction block provided by the motion compensation unit 44 from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include luma component differences or chroma component differences or both. The motion compensation unit 44 may also generate syntax elements associated with the video block of the video frame for use by the video decoder 30 when decoding the video block of the video frame. The syntax elements may include, for example, syntax elements defining a motion vector for identifying the prediction block, any flag indicating a prediction mode, or any other syntax information described herein. Note that the motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes.

[0110] In some embodiments, the intra BC unit 48 may generate vectors and extract prediction blocks in a manner similar to that described above in conjunction with the motion estimation unit 42 and the motion compensation unit 44, but these prediction blocks are in the same frame as the current block being encoded, and these vectors are referred to as block vectors rather than motion vectors. Specifically, the intra BC unit 48 may determine the intra prediction mode to be used to encode the current block. In some examples, the intra BC unit 48 may encode the current block using various intra prediction modes, for example, during different encoding passes, and test their performance through rate-distortion analysis. Next, the intra BC unit 48 may select a suitable intra prediction mode to use among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, the intra BC unit 48 may calculate rate-distortion values ​​for the various tested intra prediction modes using rate-distortion analysis, and select the intra prediction mode with the best rate-distortion characteristics among the tested modes as the suitable intra prediction mode to use. The rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block that was encoded to generate the coded block, as well as the bit rate (i.e., the number of bits) used to generate the coded block. Intra BC unit 48 may calculate ratios from the distortions and rates for the various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0111] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction in accordance with embodiments described herein. In either case, for intra block copying, the prediction block may be a block that is considered to closely match the block to be encoded in terms of pixel differences, which may be determined by SAD, SSD, or other difference metrics, and identification of the prediction block may include calculating values ​​for sub-integer pixel positions.

[0112] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, video encoder 20 can form pixel difference values ​​by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded, thereby forming a residual video block. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0113] As an alternative to the inter-frame prediction performed by the motion estimation unit 42 and the motion compensation unit 44 or the intra-frame block copy prediction performed by the intra BC unit 48 as described above, the intra-frame prediction processing unit 46 may perform intra-frame prediction on the current video block. Specifically, the intra-frame prediction processing unit 46 may determine an intra-frame prediction mode for encoding the current block. To this end, the intra-frame prediction processing unit 46 may use various intra-frame prediction modes to encode the current block, for example, during different encoding passes, and the intra-frame prediction processing unit 46 (or in some examples, the mode selection unit) may select a suitable intra-frame prediction mode from the tested intra-frame prediction modes to use. The intra-frame prediction processing unit 46 may provide information indicating the intra-frame prediction mode selected for the block to the entropy encoding unit 56. The entropy encoding unit 56 may encode the information indicating the selected intra-frame prediction mode into the bitstream.

[0114] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0115] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Optionally, entropy encoding unit 56 may perform the scan.

[0116] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as Figure 1A The video decoder 30 shown, or archived as Figure 1A Storage device 32 is shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0117] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0118] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0119] Figure 2B 30 is a block diagram showing another exemplary video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned Figure 1GThe encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0120] In some examples, units of the video decoder 30 may be tasked to perform embodiments of the present application. In addition, in some examples, embodiments of the present disclosure may be dispersed in one or more of the multiple units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in combination with other units of the video decoder 30 (such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80). In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81 (such as the motion compensation unit 82).

[0121] The video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of the video decoder 30. The video data stored in the video data memory 79 may be obtained, for example, from the storage device 32, from a local video source (such as a camera), via a wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). The video data memory 79 may include a coded picture buffer (CPB) that stores encoded video data from the encoded video bitstream. The DPB 92 of the video decoder 30 stores reference video data for use by the video decoder 30 when decoding the video data (e.g., in an intra-frame or inter-frame prediction decoding mode). The video data memory 79 and the DPB 92 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, the video data memory 79 and the DPB 92 are shown in FIG. Figure 2B 9 as two distinct components of video decoder 30. However, it will be apparent to those skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip with other components of video decoder 30, or off-chip relative to those components.

[0122] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The video decoder 30 may receive syntax elements at the video frame level and / or the video block level. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators and other syntax elements to the prediction processing unit 81.

[0123] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, intra-prediction unit 84 of prediction processing unit 81 may generate prediction data for a video block of a current video frame based on a signaled intra-prediction mode and reference data from a previously decoded block of the current frame.

[0124] When the video frame is encoded as an inter-frame prediction coded (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vectors and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks may be generated from a reference frame within one of the reference frame lists. The video decoder 30 may construct the reference frame lists, List 0 and List 1, using a default construction technique based on the reference frames stored in the DPB 92.

[0125] In some examples, when a video block is decoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0126] The motion compensation unit 82 and / or the intra BC unit 85 determine prediction information for the video block of the current video frame by parsing the motion vector and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded. For example, the motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra prediction or inter prediction) for decoding the video block of the video frame, the inter prediction frame type (e.g., B or P), the construction information for one or more of the reference frame lists for the frame, the motion vector for each inter prediction encoded video block of the frame, the inter prediction state for each inter prediction encoded video block of the frame, and other information for decoding the video block in the current video frame.

[0127] Similarly, the intra BC unit 85 may use some of the received syntax elements, such as a flag to determine whether the current video block is predicted using intra BC mode, construction information of which video blocks of the frame are within the reconstruction region and should be stored in the DPB 92, a block vector for each intra BC predicted video block of the frame, an intra BC prediction state for each intra BC predicted video block of the frame, and other information for decoding the video blocks in the current video frame.

[0128] Motion compensation unit 82 may also perform interpolation using interpolation filters as used during encoding of the video blocks by video encoder 20 to calculate interpolated values ​​for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 from the received syntax elements and use these interpolation filters to generate the prediction blocks.

[0129] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0130] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (such as a deblocking filter, an SAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video blocks in a given frame are then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for later presentation on a display device (e.g., Figure 1A on a display device 34).

[0131] In the current VVC standard and AVS3 standard, the motion information of the current coding block is copied from the spatial neighboring blocks or temporal neighboring blocks specified by the merge candidate index, or obtained through explicit signaling of motion estimation. The focus of the present disclosure is to improve the accuracy of the motion vector of the affine merge mode by improving the derivation method of the affine merge candidate. For the convenience of describing the present disclosure, the existing affine merge mode design in the VVC standard is used as an example to illustrate the proposed ideas. Please note that although the existing affine mode design in the VVC standard is used as an example throughout the present disclosure, for technicians in the field of modern video coding and decoding technology, the proposed technology can also be applied to different designs of affine motion prediction modes or other coding tools with the same or similar design spirit.

[0132] In a typical video encoding and decoding process, a video sequence usually includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other cases, a frame may be monochrome and therefore include only a two-dimensional array of luma samples.

[0133] like Figure 1C As shown in , the video encoder 20 (or more specifically, a partitioning unit in a prediction processing unit of the video encoder 20) generates an encoded representation of a frame by first partitioning the frame into a set of CTUs. A video frame may include an integer number of CTUs sequentially ordered from left to right and from top to bottom in a raster scan order. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by the video encoder 20 in a sequence parameter set so that all CTUs in a video sequence have the same size, one of 128×128, 64×64, 32×32, and 16×16. It should be noted, however, that the present application is not necessarily limited to a particular size. As Figure 1D As shown in , each CTU may include one CTB for luma samples, two corresponding coding tree blocks for chroma samples, and syntax elements for encoding and decoding samples of the coding tree blocks. The syntax elements describe the properties of different types of units of coding pixel blocks and how the video sequence can be reconstructed at the video decoder 30, including inter-frame prediction or intra-frame prediction, intra-frame prediction mode, motion vectors, and other parameters. In a monochrome picture or a picture with three separate color planes, a CTU may include a single coding tree block and syntax elements for encoding and decoding samples of the coding tree block. The coding tree block may be an N×N sample block.

[0134] To achieve better performance, video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quadtree partitioning, or a combination thereof, on the coding tree block of the CTU and divide the CTU into smaller CUs. Figure 1EAs depicted in , a 64×64 CTU 400 is first partitioned into four smaller CUs, each CU having a block size of 32×32. Among the four smaller CUs, CU 410 and CU 420 are partitioned into four CUs of a block size of 16×16, respectively. Two 16×16 CUs 430 and CU 440 are further partitioned into four CUs of a block size of 8×8, respectively. Figure 1F Depicted is a diagram showing Figure 1E The quadtree data structure of the final result of the partitioning process of CTU 400 depicted in FIG. 4 is a quadtree data structure, each leaf node of the quadtree corresponds to a CU of each size ranging from 32×32 to 8×8. Figure 1D Each CU may include a CB of luma samples and two corresponding coding blocks of chroma samples of the same size frame, and syntax elements for encoding and decoding the samples of the coding blocks. In a monochrome picture or a picture with three separate color planes, a CU may include a single coding block and syntax structures for encoding and decoding the samples of the coding block. It should be noted that Figure 1E-1F The quadtree partitioning depicted in FIG is for illustrative purposes only, and one CTU may be split into CUs based on quadtree / ternary tree / binary tree partitioning to adapt to varying local characteristics. In a multi-type tree structure, one CTU is divided by a quadtree structure, and each quadtree leaf CU may be further divided by a binary tree structure and a ternary tree structure. Figure 3A-3E As shown, there are five possible partition types for a coding block with a width of W and a height of H, namely, four-way partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal ternary partitioning, and vertical ternary partitioning.

[0135] In some embodiments, the video encoder 20 may further divide the coding block of the CU into one or more M×NPBs. A PB is a rectangular (square or non-square) sample block to which the same prediction (inter or intra) is applied. The PU of a CU may include a PB of luma samples, two corresponding PBs of chroma samples, and syntax elements for predicting the PBs. In a monochrome picture or a picture with three separate color planes, a PU may include a single PB and a syntax structure for predicting the PB. The video encoder 20 may generate a predicted luma block, a predicted Cb block, and a predicted Cr block for the luma PB, Cb PB, and Cr PB of each PU of the CU.

[0136] Video encoder 20 may use intra prediction or inter prediction to generate a prediction block for a PU. If video encoder 20 uses intra prediction to generate a prediction block for a PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of a frame associated with the PU. If video encoder 20 uses inter prediction to generate a prediction block for a PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0137] After the video encoder 20 generates the predicted luma block, the predicted Cb block, and the predicted Cr block for one or more PUs of a CU, the video encoder 20 may generate a luma residual block for the CU by subtracting the predicted luma block of the CU from the original luma coding block of the CU, so that each sample in the luma residual block of the CU indicates the difference between a luma sample in one of the predicted luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. Similarly, the video encoder 20 may generate a Cb residual block and a Cr residual block for the CU, respectively, so that each sample in the Cb residual block of the CU indicates the difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and each sample in the Cr residual block of the CU may indicate the difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.

[0138] In addition, if Figure 1E As shown in , the video encoder 20 may use quadtree partitioning to decompose the luma residual block, Cb residual block and Cr residual block of the CU into one or more luma transform blocks, Cb transform blocks and Cr transform blocks, respectively. A transform block is a rectangular (square or non-square) sample block to which the same transform is applied. A TU of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements for transforming the transform block samples. Therefore, each TU of a CU may be associated with a luma transform block, a Cb transform block and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may include a single transform block and a syntax structure for transforming the samples of the transform block.

[0139] The video encoder 20 may apply one or more transforms to the luma transform block of the TU to generate a luma coefficient block for the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficient may be a scalar. The video encoder 20 may apply one or more transforms to the Cb transform block of the TU to generate a Cb coefficient block for the TU. The video encoder 20 may apply one or more transforms to the Cr transform block of the TU to generate a Cr coefficient block for the TU.

[0140] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 may perform CABAC on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 may output a bitstream including a sequence of bits that form a representation of an encoded frame and associated data, and the bitstream is stored in storage device 32 or sent to target device 14.

[0141] After receiving the bitstream generated by the video encoder 20, the video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. The video decoder 30 may reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing the video data is generally the inverse of the encoding process performed by the video encoder 20. For example, the video decoder 30 may perform an inverse transform on a coefficient block associated with a TU of the current CU to reconstruct a residual block associated with the TU of the current CU. The video decoder 30 also reconstructs the coding block of the current CU by adding samples of the prediction block for the PU of the current CU to corresponding samples of the transform block of the TU of the current CU. After reconstructing the coding block for each CU of the frame, the video decoder 30 may reconstruct the frame.

[0142] As mentioned above, video codecs mainly use two modes, namely, intra-frame prediction (or intra-frame prediction) and inter-frame prediction (or inter-frame prediction) to achieve video compression. It should be noted that IBC can be regarded as intra-frame prediction or a third mode. Between the two modes, inter-frame prediction contributes more to codec efficiency than intra-frame prediction because motion vectors are used to predict the current video block from the reference video block.

[0143] However, with the ever-improving video data capture technology and more refined video block sizes for retaining details in the video data, the amount of data required to represent the motion vector of the current frame has also increased significantly. One way to overcome this challenge is to benefit from the fact that not only a group of neighboring CUs in both the spatial domain and the temporal domain have similar video data for prediction purposes, but the motion vectors between these neighboring CUs are also similar. Therefore, the motion information of spatially neighboring CUs and / or temporally co-located CUs can be used as an approximation of the motion information (e.g., motion vector) of the current CU by exploring the spatial and temporal correlation of the spatially neighboring CUs and / or the temporally co-located CUs, which is also referred to as the "motion vector predictor (MVP)" of the current CU.

[0144] Instead of combining as above Figure 1B The actual motion vector of the current CU determined by the motion estimation unit is encoded into the video bitstream, and the motion vector prediction factor of the current CU is subtracted from the actual motion vector of the current CU to generate the motion vector difference (MVD) of the current CU. By doing so, the motion vector determined by the motion estimation unit for each CU of the frame does not need to be encoded into the video bitstream, and the amount of data used to represent motion information in the video bitstream can be significantly reduced.

[0145] As in the process of selecting a prediction block in a reference frame during inter-frame prediction of a coding block, both the video encoder 20 and the video decoder 30 need to adopt a set of rules for constructing a motion vector candidate list (also called a "merge list") of the current CU using those potential candidate motion vectors associated with the spatial neighboring CUs and / or the temporal co-located CUs of the current CU, and then selecting a member from the motion vector candidate list as the motion vector predictor of the current CU. By doing so, the motion vector candidate list itself does not need to be sent from the video encoder 20 to the video decoder 30, and the index of the selected motion vector predictor in the motion vector candidate list is sufficient for the video encoder 20 and the video decoder 30 to use the same motion vector predictor in the motion vector candidate list to encode and decode the current CU.

[0146] Affine Model

[0147] In HEVC, only the translational motion model is applied for motion compensated prediction. In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC and AVS3, affine motion compensated prediction is applied by signaling a flag for each inter-frame coding block to indicate whether the translational motion model or the affine motion model is applied for inter-frame prediction. In the current VVC and AVS3 design, two affine modes are supported for one affine coding block, including a 4-parameter affine mode and a 6-parameter affine mode.

[0148] The 4-parameter affine model has the following parameters: two parameters for horizontal translation and vertical translation, one parameter for scaling, and one parameter for rotation in both directions. In this model, the horizontal scaling parameter is equal to the vertical scaling parameter, and the horizontal rotation parameter is equal to the vertical rotation parameter. To achieve a better adaptation of the motion vectors and affine parameters, those affine parameters are derived from two MVs (also called control point motion vectors (CPMVs)) located at the top left and top right corners of the current block. Figure 4A-4BAs shown, the affine motion field of the block consists of two CPMVs (V 0 ,V 1 ) description. Based on the motion of the control points, the motion field (v x ,v y ) is described as:

[0149]

[0150] The 6-parameter affine model has the following parameters: two parameters for horizontal translation and vertical translation, two parameters for horizontal scaling and rotation, and two parameters for vertical scaling and rotation. The 6-parameter affine motion model uses three CPMV codecs. Figure 5 As shown in , the three control points of a 6-parameter affine block are located at the upper left corner, upper right corner, and lower left corner of the block. The motion at the upper left control point is related to the translation motion, the motion at the upper right control point is related to the rotation motion and scaling motion in the horizontal direction, and the motion at the lower left control point is related to the rotation motion and scaling motion in the vertical direction. Compared with the 4-parameter affine motion model, the rotation motion and scaling motion in the horizontal direction of the 6-parameter can be different from those in the vertical direction. Assume (V 0 ,V 1 ,V 2 )yes Figure 5 The MV of the upper left corner, the MV of the upper right corner, and the MV of the lower left corner of the current block in , and the three MVs at the control point are used to derive each sub-block (v x ,v y ) is:

[0151]

[0152] Affine merge mode

[0153] In affine merge mode, the CPMV for the current block is not explicitly signaled, but derived from neighboring blocks. Specifically, in this mode, the motion information of spatially neighboring blocks is used to generate the CPMV for the current block. The affine merge mode candidate list has a finite size. For example, in the current VVC design, there may be up to five candidates. The encoder can evaluate and select the best candidate index based on a rate-distortion optimization algorithm. The selected candidate index is then signaled to the decoder side. Affine merge candidates can be determined in three ways. In the first way, affine merge candidates can be inherited from neighboring affine encoded blocks. In the second way, affine merge candidates can be constructed from translation MVs from neighboring blocks. In the third way, zero MV is used as an affine merge candidate.

[0154] For the inheritance method, there may be up to two candidates. From the neighboring block located to the lower left of the current block (e.g., Figure 6 As shown in FIG, the scanning order is from A0 to A1) and from the neighboring block located to the upper right of the current block (for example, Figure 6 As shown in , the scanning order is from B0 to B2) (if the neighboring blocks are available) to obtain candidates.

[0155] For the constructed method, a candidate is a combination of the translated MVs of the neighbors that can be generated by two steps.

[0156] Step 1: Get four translation MVs from available neighbors, including MV1, MV2, MV3, and MV4.

[0157] MV1: The MV from one of the three neighboring blocks near the upper left corner of the current block. Figure 7 As shown, the scanning order is B2, B3 and A2.

[0158] MV2: The MV from one of the two neighboring blocks near the upper right corner of the current block. Figure 7 As shown, the scanning order is B1 and B0.

[0159] MV3: The MV from one of the two neighboring blocks near the lower left corner of the current block. Figure 7 As shown, the scanning order is A1 and A0.

[0160] MV4: The MV of the temporally co-located block from the neighboring block near the lower right corner of the current block. Figure 7 As shown in , the neighboring blocks are T.

[0161] Step 2: Derive a combination based on the four translation MVs from step 1.

[0162] Combination 1: MV1, MV2, MV3;

[0163] Combination 2: MV1, MV2, MV4;

[0164] Combination 3: MV1, MV3, MV4;

[0165] Combination 4: MV2, MV3, MV4;

[0166] Combination 5: MV1, MV2;

[0167] Combination 6: MV1, MV3.

[0168] When the merge candidate list is not full after being populated with inherited candidates and constructed candidates, a zero MV is inserted at the end of the list.

[0169] Affine AMVP mode

[0170] Affine advanced motion vector prediction (AMVP) mode is applicable to CUs with both width and height greater than or equal to 16. A CU-level affine flag is signaled in the bitstream to indicate whether the affine AMVP mode is used, and then another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMV of the current CU and its CPMV predictor (CPMVP) is signaled in the bitstream. The affine AVMP candidate list size is 2, and the affine AMVP candidate list is generated by using the following four types of CPMV candidates in the following order:

[0171] —Inherited affine AMVP candidates derived from the CPMV of neighboring CUs;

[0172] - Constructed affine AMVP candidate (CPMVP) derived using the translated MV of the neighboring CU;

[0173] - translation MV from neighboring CU;

[0174] - Temporal MV from the co-located CU; and

[0175] —Zero MV.

[0176] The order of checking for inherited affine AMVP candidates is the same as that for inherited affine merge candidates. The only difference is that for AMVP candidates, only affine CUs with the same reference picture as the current block are considered. When the inherited affine motion predictor is inserted into the candidate list, the pruning process is not applied.

[0177] Construct AMVP candidates derived from the same spatial neighbors as in affine merge mode. Use the same check order as used in affine merge candidate construction. In addition, check the reference picture index of neighboring blocks. Use the first block in the check order that is inter-coded and has the same reference picture as the current CU. When the current CU is coded with 4-parameter affine mode and mv 0 and mv 1 When both are available, mv 0 and mv 1 Add as a candidate in the affine AMVP candidate list. When the current CU is encoded with 6-parameter affine mode and all three CPMVs are available, add them as a candidate in the affine AMVP candidate list. Otherwise, set the constructed AMVP candidate to unavailable.

[0178] If after inserting valid inherited affine AMVP candidates and building AMVP candidates, the number of candidates in the affine AMVP list is still less than 2, then mv will be added in order 0 、mv1 and mv 2 As the translation MV to predict all control point MVs of the current CU (if available). Finally, if the affine AMVP list is still not full, the affine AMVP list is filled with zero MVs.

[0179] Regular inter-frame merging mode

[0180] In some embodiments, a conventional inter-merge candidate list is constructed by including the following five types of candidates in order:

[0181] (1) Spatial MVP from spatially adjacent CUs;

[0182] (2) Temporal MVP from the co-located CU;

[0183] (3) history-based MVP from a first-in, first-out (FIFO) table;

[0184] (4) Pairwise average MVP; and

[0185] (4) Zero MV.

[0186] The size of the merge list is signaled in the sequence parameter set header, and the maximum allowed size of the merge list is 6. For each CU code in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is context coded, and bypass coding is used for the other bins.

[0187] The above provides a process for deriving merge candidates for each category. In some embodiments, for all CUs in a region of a certain size, parallel derivation of merge candidate lists may be supported.

[0188] Spatial candidate export

[0189] The derivation of spatial merge candidates in VVC is the same as in HEVC, except that the positions of the first two merge candidates are swapped. Figure 4C A maximum of four merge candidates are selected from the candidates in the positions depicted in . The derived order is B0, A0, B1, A1 and B2. Position B2 is considered only when one CU or more than one CU in positions B0, A0, B1, A1 is not available (for example, because it belongs to another slice or tile) or is intra-coded. After adding the candidate at position A1, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving the encoding and decoding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only pairs with Figure 4DThe pairs linked by arrows in , and only add the candidate to the list if the corresponding candidate for redundancy check does not have the same motion information. Figure 4D Candidate pairs considered for redundancy check of spatial merging candidates are shown.

[0190] Temporal Candidate Export

[0191] In this step, only one candidate is added to the list. Specifically, in the derivation of this temporal merge candidate, the scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list and reference index used to derive the co-located CU are explicitly signaled in the slice header. Figure 4E As shown by the dotted line in , a scaled motion vector for the temporal merge candidate is obtained, which is scaled from the motion vector of the co-located CU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero.

[0192] In Candidate C 0 and candidate C 1 Select the location of the time candidate between Figure 4F If position C 0 If the CU at position C is not available, is intra-coded, or is outside the current row of the CTU, then position C is used. 1 Otherwise, use position C when deriving temporal merge candidates. 0 .

[0193] History-based merge candidate export

[0194] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and temporal motion vector prediction (TMVP). In this method, the motion information of the previously coded block is stored in a table and used as the MVP for the current CU. A table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (cleared) when a new CTU row is encountered. Whenever there is a non-subblock inter-coded CU, the associated motion information is added to the last entry of the table as a new HMVP candidate.

[0195] The HMVP table size S can be set to 6, which indicates that up to 5 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is used, where a redundancy check is first applied to find if the same HMVP exists in the table. If found, the same HMVP is removed from the table, and all subsequent HMVP candidates are moved forward, and the same HMVP is inserted into the last entry of the table.

[0196] HMVP candidates can be used in the merge candidate list construction process. The last few HMVP candidates in the table are checked in order and inserted into the candidate list after the TMVP candidate. HMVP candidates apply redundancy check to spatial merge candidates or temporal merge candidates.

[0197] In order to reduce the number of redundant check operations, the following simplifications are introduced. First, the last two entries in the table perform redundant checks on the A1 and B1 spatial candidates, respectively. Second, once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the process of building a merge candidate list from the HMVP is terminated.

[0198] Pairwise average merge candidate derivation

[0199] A pairwise average candidate is generated by averaging the predefined candidate pairs in the existing merge candidate list using the first two merge candidates. The first merge candidate may be defined as p0Cand, and the second merge candidate may be defined as p1Cand. For each reference list, the average motion vector is calculated according to the availability of motion vectors p0Cand and p1Cand, respectively. If two motion vectors are available in one list, they are averaged even when the two motion vectors point to different reference pictures, and their reference pictures are set to the reference pictures of p0Cand; if only one motion vector is available, the motion vector is used directly; if no motion vector is available, the list is kept invalid. In addition, if the half-pixel interpolation filter indexes of p0Cand and p1Cand are different, they are set to 0.

[0200] When the merge list is not full after adding pairwise average merge candidates, zero MVPs are inserted at the end of the merge list until the maximum number of merge candidates is met.

[0201] Adaptive Reordering of MergeCandidates with Template Matching (ARMC)

[0202] The reordering method named ARMC is applied to normal merge mode, template matching (TM) merge mode and affine merge mode (excluding SbTMVP candidates), where SbTMVP represents sub-block based temporal motion vector prediction candidates. For TM merge mode, merge candidates are reordered before refinement process.

[0203] After constructing the merge candidate list, the merge candidates are divided into several subgroups. The subgroup size is set to 5. The merge candidates in each subgroup are reordered in ascending order according to the cost value based on template matching. For simplicity, the merge candidates in the last subgroup (not the first subgroup) are not reordered.

[0204] The template matching cost is measured by the sum of absolute differences (SAD) between the samples of the template of the current block and its corresponding reference samples. The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are located by the same motion information of the current block.

[0205] When the merge candidate uses bidirectional prediction, the reference samples of the merge candidate template are also obtained by Fig.19 The bidirectional prediction shown is generated.

[0206] Regression-based affine candidate derivation

[0207] In the VVC standardization process, a regression-based motion vector field (RMVF) derivation method was proposed in JVET-L0171, which provides multiple new sub-block-based merge candidates. Fig.32 As shown, the motion vectors and center positions of neighboring sub-blocks from the current CU are used as input to a linear regression process to derive a set of linear model parameters.

[0208] In the JVET standardization exploration process after VVC, a regression-based affine candidate derivation method was proposed in JVET-Z0125. In this method, the sub-block motion field from the previously encoded affine CU and the motion vectors of the neighboring sub-blocks from the current CU are used as inputs to the regression process. Compared with the regression process in JVET-L0171, the difference is that the predicted CPMV is derived as output instead of the sub-block motion field for the current block. The derived CPMV can be added to the sub-block merge candidate list or the affine AMVP list.

[0209] merge mode with MVD (MMVD)

[0210] In addition to the merge mode (in which the implicitly derived motion information is directly used for prediction sample generation of the current CU), a merge mode with motion vector difference (MMVD) is introduced in VVC. Immediately after sending the normal merge flag, the MMVD flag is signaled to indicate whether the MMVD mode is used for the CU.

[0211] Under MMVD, after a merge candidate is selected, the merge candidate is further refined by signaling MVD information. Further information includes a merge candidate flag, an index for specifying the magnitude of motion, and an index for indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. An MMVD candidate flag is signaled to specify which one to use between the first and second merge candidates.

[0212] The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Fig.26 As shown, the offset is added to either the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 6 below.

[0213] Table 6

[0214]

[0215] The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent four directions as shown in the table above. It should be noted that the meaning of the MVD symbol may change depending on the information of the starting MV. When the starting MV is a unidirectional prediction MV or a bidirectional prediction MV in which both lists point to the same side of the current picture (i.e., the POCs of both references are greater than the POC of the current picture, or are less than the POC of the current picture), the symbol in the following Table 7 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV in which the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture and the POC of the other reference is less than the POC of the current picture), and the difference in POC in list 0 is greater than the difference in POC in list 1, the symbol in the following table specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign for list 1 MV has an opposite value. Otherwise, if the difference of the POC in List 1 is greater than List 0, then the sign in the following table specifies the sign of the MV offset added to the List 1 MV component of the starting MV, and the sign of the List 0 MV has the opposite value.

[0216] The MVD is scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is greater than the POC difference in list 1, the MVD of list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb, as Fig.23 If the POC difference of L1 is greater than L0, the MVD of list 0 is scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MV.

[0217] Table 7

[0218] Direction Index 00 01 10 11 x-axis + - N / A N / A y-axis N / A N / A + -

[0219] For a sub-block based merge candidate with a sub-block size of Wsub×Hsub, the upper template includes several sub-templates of size Wsub×1, and the left template includes several sub-templates of size 1×Hsub. Wsub is the width of the sub-block, and Hsub is the height of the sub-block. Fig. 20 As shown, the motion information of the sub-blocks in the first row and the first column of the current block is used to derive the reference samples of each sub-template.

[0220] For the current video standards VVC and AVS, only adjacent neighboring blocks are used to derive affine merge candidates for the current block, such as Figure 6 and Figure 7 In order to increase the diversity of merged candidates and further explore spatial correlation, the coverage of neighboring blocks is directly extended from adjacent areas to non-adjacent areas.

[0221] In the current video standards VVC and AVS, each affine inheritance candidate is derived from one neighboring block with affine motion information. On the other hand, each affine construction candidate is derived from two or three neighboring blocks with translational motion information. To further explore spatial correlation, new candidate derivation methods that combine affine motion and translational motion can be studied.

[0222] The candidate derivation method proposed for affine merge mode can be extended to other coding modes, such as affine AMVP mode and regular merge mode.

[0223] In the present disclosure, the candidate derivation process for the affine merge mode is extended by using not only adjacent neighboring blocks but also non-adjacent neighboring blocks. The detailed method can be summarized in the following aspects, including affine merge candidate pruning, non-adjacent neighbor-based derivation process for affine inheritance merge candidates, non-adjacent neighbor-based derivation process for affine construction merge candidates, inheritance-based derivation method for affine construction merge candidates, HMVP-based derivation method for affine construction merge candidates, candidate derivation method for affine AMVP mode and conventional merge mode, and motion information storage.

[0224] Affine Merge Candidate Pruning

[0225] Since the affine merge candidate list in typical video codec standards usually has a limited size, candidate pruning is a necessary process to remove redundant candidates. This pruning process is required for both affine merge inheritance candidates and construction candidates. As explained in the introduction, the CPMV of the current block is not directly used for affine motion compensation. Instead, the CPMV needs to be converted to a translation MV at the location of each sub-block within the current block. The conversion process is performed by following the general affine model as shown below:

[0226]

[0227] where (a, b) are the differential translation parameters, (c, d) are the incremental scaling and rotation parameters for the horizontal direction, (e, f) are the incremental scaling and rotation parameters for the vertical direction, and (x, y) are the horizontal and vertical distances of the sub-block's pivot position (e.g., center or top-left corner) relative to the top-left corner of the current block (e.g., Figure 5 The coordinates (x, y) shown), and (v x ,v y ) is the target translation MV of the sub-block.

[0228] For the 6-parameter affine model, the three CPMVs (called V 0 、V 1 and V 2 ) is available. Then the six model parameters a, b, c, d, e and f can be calculated as:

[0229]

[0230] For the 4-parameter affine model, if the upper left corner CPMV and the upper right corner CPMV (called V 0 and V 1 ) is available, then the six parameters a, b, c, d, e and f can be calculated as:

[0231]

[0232] For the 4-parameter affine model, if the upper left corner CPMV and the lower left corner CPMV (called V 0 and V 2 ) is available, then the six parameters a, b, c, d, e and f can be calculated as:

[0233]

[0234] In the above equations (4), (5) and (6), w and h represent the width and height of the current block, respectively.

[0235] When comparing two CPMV merging candidate sets for redundancy check, it is proposed to check the similarity of 6 affine model parameters. Therefore, the candidate pruning process can be performed in two steps.

[0236] In step 1, given two CPMV candidate sets, the corresponding affine model parameters for each candidate set are derived. More specifically, the two CPMV candidate sets can be represented by two affine model parameter sets, for example (a 1 ,b 1 ,c 1 ,d 1 ,e 1 ,f 1 ) and (a 2 ,b 2 c 2 ,d 2 , e 2 ,f 2 ).

[0237] In step 2, a similarity check is performed between two sets of affine model parameters based on one or more predefined thresholds. 1 -a 2 )、(b 1 -b 2 )、(c 1 -c 2 )、(d 1 -d 2 )、(e 1 -e 2 ) and (f 1 -f 2 ) are both below a positive threshold, such as a value of 1, the two candidates are considered similar and one of them may be pruned / removed and not included in the merge candidate list.

[0238] In some embodiments, the division operation or right shift operation in step 1 may be removed to simplify the calculations in the CPMV pruning process.

[0239] Specifically, the model parameters c, d, e and f can be calculated without dividing by the width w and height h of the current block. For example, taking the above equation (4) as an example, the approximate model parameters c', d', e' and f' can be calculated as the following equation (7).

[0240]

[0241] In the case where only two CPMVs are available, one part of the model parameters is derived from the other part of the model parameters, which depends on the width or height of the current block. In this case, the model parameters can be converted to take into account the influence of width and height. For example, in the case of equation (5), the approximate model parameters c', d', e' and f' can be calculated based on the following equation (8). In the case of equation (6), the approximate model parameters c', d', e' and f' can be calculated based on the following equation (9).

[0242]

[0243] When the approximate model parameters c′, d′, e′, and f′ are calculated in step 1 above, the calculation of the absolute value required for the similarity check in step 2 above may be changed accordingly: (a 1 -a 2 )、(b 1 -b 2 ), (c′ 1 -c′ 2 ), (d′ 1 -d′ 2 ), (e′ 1 -e′ 2 ) and (f′ 1 -f′ 2 ).

[0244] In step 2 above, a threshold is required to evaluate the similarity between two CPMV candidate sets. There are many ways to define the threshold. In one embodiment, a threshold can be defined for each comparable parameter. Table 1 is an example in this embodiment, showing the threshold defined for each comparable model parameter. In another embodiment, the threshold can be defined by considering the size of the current coding block. Table 2 is an example in this embodiment, showing the threshold defined by the size of the current coding block.

[0245] Table 1

[0246] Comparable parameters Threshold a 1 b 1 c 2 d 2 e 2 f 2

[0247] Table 2

[0248] The size of the current block Threshold Size <= 64 pixels 1 64 pixels < size <= 256 pixels 2 256 pixels < size <= 1024 pixels 4 1024 pixels < size 8

[0249] In another embodiment, the threshold value may be defined by considering the weight or height of the current block. Table 3 and Table 4 are examples in this embodiment. Table 3 shows the threshold value defined by the width of the current coding block, and Table 4 shows the threshold value defined by the height of the current coding block.

[0250] Table 3

[0251] The width of the current block Threshold Width <= 8 pixels 1 8 pixels < width <= 32 pixels 2 32 pixels < width <= 64 pixels 4 64 pixels < width 8

[0252] Table 4

[0253] The current block height Threshold Height <= 8 pixels 1 8 pixels < height <= 32 pixels 2 32 pixels < height <= 64 pixels 4 64 pixels < height 8

[0254] In another embodiment, the threshold value may be defined as a set of fixed values. In another embodiment, the threshold value may be defined by any combination of the above embodiments. In one example, the threshold value may be defined by considering different parameters and the weight and height of the current block. Table 5 is an example in this embodiment, showing the threshold value defined by the height of the current coding block. Note that in any of the above-mentioned embodiments, where necessary, the comparable parameter may represent any parameter defined in any equation from equation (4) to equation (9).

[0255] Table 5

[0256]

[0257] The benefits of using the transformed affine model parameters for candidate redundancy checking include: it creates a unified similarity checking process for candidates with different affine model types, for example, one merge candidate may use a 6-parameter affine model with three CPMVs, while another candidate may use a 4-parameter affine model with two CPMVs; it considers the different impact of each CPMV in the merge candidate when deriving the target MV at each sub-block; and it provides similar importance of two affine merge candidates with respect to the width and height of the current block.

[0258] Non-adjacent neighbor-based derivation process for affine inheritance merge candidates

[0259] For the inherited merge candidates, the non-adjacent neighbor-based derivation process can be performed in three steps: Step 1 is for candidate scanning. Step 2 is for CPMV projection. Step 3 is for candidate pruning.

[0260] In step 1, non-adjacent neighboring blocks are scanned and selected by the following method.

[0261] Scanning area and distance

[0262] In some examples, non-adjacent neighboring blocks may be scanned from the left area and the upper area of ​​the current coding block. The scanning distance may be defined as the number of coding blocks from the scanning position to the left or upper side of the current coding block.

[0263] like Figure 8 As shown in , multiple non-adjacent neighboring block lines can be scanned on the left or above the current coding block. Figure 8 The distance shown indicates the number of coding blocks from each candidate position to the left or top of the current block. For example, an area with "Distance 2 (D2)" on the left side of the current block indicates that the candidate neighboring block located in the area is 2 blocks away from the current block. Similar indications can be applied to other scanning areas with different distances.

[0264] In one or more embodiments, the non-adjacent neighboring blocks at each distance may have the same block size as the current coding block, such as Fig.13A As shown. Fig.13A As shown, the non-adjacent neighboring block 1301 on the left and the non-adjacent neighboring block 1302 on the upper side have the same size as the current block 1303. In some embodiments, the non-adjacent neighboring blocks at each distance may have a different block size from the current coding block, such as Fig. 13B As shown. The neighboring block 1304 is a neighboring block adjacent to the current block 1303. Fig. 13B As shown, the non-adjacent neighboring block 1305 on the left and the non-adjacent neighboring block 1306 on the upper side have the same size as the current block 1307. The neighboring block 1308 is a neighboring block adjacent to the current block 1307.

[0265] Note that, in the case where the non-adjacent neighboring blocks at each distance have the same block size as the current coding block, the value of the block size is adaptively changed according to the division granularity at each different area in the image. Note that when the non-adjacent neighboring blocks at each distance have a different block size from the current coding block, the value of the block size may be predefined as a constant value such as 4×4, 8×8, or 16×16. Fig.10 and Fig.12 The 4x4 non-adjacent motion fields shown in are examples of such cases, where motion fields may be considered as, but not limited to, a special case of sub-blocks.

[0266] Similarly, Fig.11 The non-adjacent coding blocks shown in may also have different sizes. In one example, the non-adjacent coding blocks may have the same size as the current coding block, which is adaptively changed. In another example, the non-adjacent coding blocks may have a predefined size, which has a fixed value, such as 4×4, 8×8, or 16×16.

[0267] Based on the defined scanning distance, the total size of the scanning area on the left or above the current coding block can be determined by a configurable distance value. In one or more embodiments, the maximum scanning distances on the left and top sides may use the same value or different values. Figure 13 shows an example in which the maximum distances on both the left and top sides share the same value of 2. The maximum scanning distance value may be determined by the encoder side and signaled in the bitstream. Alternatively, the maximum scanning distance value may be predefined as a fixed value, such as a value of 2 or 4. When the maximum scanning distance is predefined as a value of 4, it indicates that the scanning process terminates when the candidate list is full or all non-adjacent neighboring blocks with a maximum distance of 4 have been scanned, whichever comes first.

[0268] In one or more embodiments, within each scanning region at a specific distance, the start neighboring block and the end neighboring block may be position-dependent.

[0269] In some embodiments, for the left scanning area, the starting adjacent block may be an adjacent lower left block of the starting adjacent block of the adjacent scanning area with a smaller distance. Figure 8 As shown in FIG. 1 , the starting neighboring block of the “Distance 2” scanning area on the left side of the current block is the lower left neighboring block of the starting neighboring block of the “Distance 1 (D1)” scanning area. Figure 8 In FIG. 1 , D1, D2, and D3 represent distance 1, distance 2, and distance 3, respectively. The end neighboring block may be a left block adjacent to the end neighboring block of the upper scanning area having a smaller distance. Figure 8 As shown, the end neighboring block of the "Distance 2" scanning area on the left side of the current block is the adjacent left neighboring block of the end neighboring block of the "Distance 1" scanning area above the current block.

[0270] Similarly, for the upper scanning area, the start neighboring block may be the upper right block adjacent to the start neighboring block of the adjacent scanning area with a smaller distance, and the end neighboring block may be the upper left block adjacent to the end neighboring block of the adjacent scanning area with a smaller distance.

[0271] Scanning order

[0272] When scanning neighboring blocks in a non-adjacent area, a particular order and / or rule may be followed to determine the selection of neighboring blocks to be scanned.

[0273] In some embodiments, the left area may be scanned first, and then the upper area may be scanned. Figure 8 As shown, three lines of non-adjacent areas on the left (eg, from distance 1 (D1) to distance 3 (D3)) may be scanned first, and then three lines of non-adjacent areas above the current block may be scanned.

[0274] In some embodiments, the left area and the upper area may be scanned alternately. Figure 8As shown, the left scanning area with "Distance 1" is scanned first, and then the upper area with "Distance 1" is scanned.

[0275] For scanning areas located on the same side (e.g., the left area or the upper area), the scanning order is from the area with a small distance to the area with a large distance. This order can be flexibly combined with other embodiments of the scanning order. For example, the left area and the upper area can be scanned alternately, and the order for the same side area is scheduled from a small distance to a large distance.

[0276] In each scanning area at a specific distance, a scanning order can be defined. In one embodiment, for the left scanning area, scanning can start from the bottom adjacent block to the top adjacent block. For the upper scanning area, scanning can start from the right block to the left block.

[0277] Scan termination

[0278] For the inherited merge candidate, the neighboring blocks encoded with the affine mode are defined as qualified candidates. In some embodiments, the scanning process can be performed interactively. For example, when the first X qualified candidates are identified, the scanning performed in a specific area at a specific distance can be stopped, where X is a predefined positive value. For example, Figure 8 As shown, when the first one or more qualified candidates are identified, scanning in the left scanning region with distance 1 may be stopped. Then, the next iteration of the scanning process is started by targeting another scanning region, which is dictated by the predefined scanning order / rule.

[0279] In one or more embodiments, X may be defined for each distance. For example, at each distance, X is set to 1, which means that if the first qualified candidate is found, the scan is terminated for each distance, and the scanning process is restarted from a different distance of the same area or the same or different distance of a different area. Note that the value of X may be set to the same value or different values ​​for different distances. If the maximum number of qualified candidates is found from all allowable distances of the area (e.g., constrained by a maximum distance), the scanning process for a region is completely terminated.

[0280] In another embodiment, X may be defined for a region. For example, X is set to 3, which means that if the first three qualified candidates are found, the scanning for the entire region (e.g., the left region or the upper region of the current block) is terminated, and the scanning process is restarted from the same or different distance of another region. Note that for different regions, the value of X may be set to the same value or different values. If the maximum number of qualified candidates is found from all regions, the entire scanning process is completely terminated.

[0281] The value of X may be defined for both distance and region. For example, for each region (e.g., the left region or the upper region of the current block), X is set to 3, and for each distance, X is set to 1. For different regions and distances, the value of X may be set to the same value or different values.

[0282] In some embodiments, the scanning process may be performed continuously. For example, the scanning performed in a specific area at a specific distance may be stopped when all covered neighboring blocks have been scanned and no more qualified candidates are identified or the maximum allowable number of candidates is reached.

[0283] During the candidate scanning process, each candidate non-adjacent neighboring block is determined and scanned by following the scanning method proposed above. For easier implementation, each candidate non-adjacent neighboring block can be indicated or located by a specific scanning position. Once the specific scanning area and scanning distance are determined by following the method proposed above, the scanning position can be determined accordingly based on the following method.

[0284] In one method, the lower left position and the upper right position are used for the upper non-adjacent neighboring block and the left non-adjacent neighboring block, respectively, such as Fig.15A as shown in .

[0285] In another method, the lower right position is used for both the upper non-adjacent neighboring block and the left non-adjacent neighboring block, such as Fig. 15B as shown in .

[0286] In another method, the lower left position is used for both the upper non-adjacent neighboring block and the left non-adjacent neighboring block, such as Fig. 15C as shown in .

[0287] In another method, the upper right position is used for both the upper non-adjacent neighboring block and the left non-adjacent neighboring block, such as Fig.15D as shown in .

[0288] To make it easier to explain, Figures 15A-15D In , it is assumed that each non-adjacent neighboring block has the same block size as the current block. Without loss of generality, this description can be easily extended to non-adjacent neighboring blocks with different block sizes.

[0289] In addition, in step 2, the same CPMV projection process used in the current AVS standard and VVC standard can be used. In this CPMV projection process, assuming that the current block shares the same affine model with the selected neighboring block, two or three corner pixel coordinates are substituted into equation (1) or (2) (for example, if the current block uses a 4-parameter model, two coordinates (upper left pixel / sample position and upper right pixel / sample position) are used; if the current block uses a 6-parameter model, three coordinates (upper left pixel / sample position, upper right pixel / sample position, and lower left pixel / sample position) are used), depending on whether the neighboring block is encoded using a 4-parameter affine model or a 6-parameter affine model to generate two or three CPMVs.

[0290] In step 3, any qualified candidate identified in step 1 and converted in step 2 may be checked for similarity against all existing candidates already in the merge candidate list. The details of the similarity check have been described in the "Affine Merge Candidate Pruning" section above. If a new qualified candidate is found to be similar to any existing candidate in the candidate list, the new qualified candidate is removed / pruned.

[0291] Non-adjacent neighbor-based derivation process for affine constructed merge candidates

[0292] In the case of deriving inherited merge candidates, one neighboring block is identified at a time, where this single neighboring block needs to be encoded in affine mode and may contain two or three CPMVs. In the case of deriving constructed merge candidates, two or three neighboring blocks may be identified at a time, where each identified neighboring block does not need to be encoded in affine mode, and only one translation MV is retrieved from the block.

[0293] Fig. 9 An example is presented in which affine merge candidates can be derived by using non-adjacent neighboring blocks. Fig. 9 In the example, A, B, and C are the area positions of three non-adjacent neighboring blocks. A virtual coding block is formed by using the position of A as the upper left corner, the position of B as the upper right corner, and the position of C as the lower left corner. If the virtual CU is regarded as an affine coding block, the MVs at the positions of A', B', and C' can be derived by following equation (3), where the model parameters (a, b, c, d, e, f) can be calculated by the translation MVs at the positions of A, B, and C. Once derived, the MVs at the positions of A', B', and C' can be used as three CPMVs for the current block, and the existing process for generating and constructing affine merge candidates (the process used in the AVS standard and the VVC standard) can be used.

[0294] For constructing merge candidates, the derivation process based on non-adjacent neighbors can be performed in five steps. The derivation process based on non-adjacent neighbors can be performed in five steps in a device such as an encoder or a decoder. Step 1 is used for candidate scanning. Step 2 is used for affine model determination. Step 3 is used for CPMV projection. Step 4 is used for candidate generation. And step 5 is used for candidate pruning. In step 1, non-adjacent neighboring blocks can be scanned and selected by the following method.

[0295] Scanning area and scanning distance

[0296] In some embodiments, in order to maintain a rectangular coding block, the scanning process is performed only for two non-adjacent neighboring blocks.The third non-adjacent neighboring block may depend on the horizontal position and the vertical position of the first non-adjacent neighboring block and the second non-adjacent neighboring block.

[0297] In some embodiments, Fig. 9 As shown, the scanning process is performed only on the positions of B and C. The position of A can be uniquely determined by the horizontal position of C and the vertical position of B.

[0298] In order to form a valid virtual coding block, at least the position of A may need to be valid. The validity of position A may be defined as whether motion information at position A is available. In one embodiment, the coding block at position A may need to be encoded in inter-frame mode so that motion information can be used to form the virtual coding block.

[0299] In some embodiments, the scanning area and the scanning distance may be defined according to a specific scanning direction.

[0300] In some embodiments, the scanning direction may be perpendicular to a side of the current block. Fig.10 An example is shown in , where the scan area is defined as a line of continuous motion fields to the left or above the current block. The scan distance is defined as the number of motion fields from the scan position to one side of the current block. Note that the size of the motion field may depend on the maximum granularity of the applicable video codec standard. Fig.10 In the example shown, the size of the motion field is assumed to be aligned with the current VVC standard and is set to 4×4.

[0301] In some embodiments, the scan direction may be parallel to a side of the current block. Fig.11 An example is shown in , where the scan area is defined as a line of consecutive coded blocks to the left or above the current block.

[0302] In some embodiments, the scanning direction may be a combination of vertical scanning and parallel scanning to one side of the current block. Fig.12 An example is shown in FIG. Fig.12As shown, the scanning direction can also be a combination of parallel and diagonal. The scan at position B starts from left to right and then goes diagonally to the upper left block. The scan at position B will be as follows Fig.12 Repeat as shown. Similarly, the scan at position C starts from top to bottom and then goes diagonally to the upper left block. The scan at position C will be as follows Fig.12 Repeat as shown.

[0303] Scanning order

[0304] In some embodiments, the scanning order may be defined as from a position having a smaller distance from the current coding block to a position having a larger distance from the current coding block. This order may be applied to the case of vertical scanning.

[0305] In some embodiments, the scanning order may be defined as a fixed pattern. This fixed pattern scanning order may be used for candidate positions with similar distances. An example is the case of parallel scanning. In one example, the scanning order may be defined as a top-down direction for the left scanning area, and may be defined as a left-to-right direction for the upper scanning area, such as Fig.11 The example shown in .

[0306] In the case of a combined scanning method, the scanning sequence can be a combination of fixed pattern and distance-dependent, e.g. Fig.12 Example shown.

[0307] Scan termination

[0308] For constructing merge candidates, eligible candidates do not need to be affine coded since only the MV needs to be translated.

[0309] Depending on the number of candidates required, the scanning process may be terminated when the first X qualified candidates are identified, where X is a positive value.

[0310] like Fig. 9 As shown, in order to form a virtual coding block, three corners named A, B and C are required. For easier implementation, only the scanning process in step 1 can be performed to identify non-adjacent neighboring blocks located at corners B and C, and the coordinates of A can be accurately determined by taking the horizontal coordinates of C and the vertical coordinates of B. In this way, the virtual coding block formed is restricted to a rectangle. In the case where point B or point C is not available (e.g., outside the boundary) or the motion information at the non-adjacent neighboring blocks corresponding to B or C is not available (e.g., the block is encoded in intra mode or screen content mode), the horizontal coordinate or vertical coordinate of C can be defined as the horizontal coordinate or vertical coordinate of the upper left point of the current block, respectively.

[0311] In another embodiment, when angle B and / or angle C are first determined from the scanning process in step 1, non-adjacent neighboring blocks located at angle B and / or angle C may be identified accordingly. Second, the position of angle B and / or angle C may be reset to a pivot point within the corresponding non-adjacent neighboring block, such as the centroid of each non-adjacent neighboring block. For example, the centroid may be defined as the geometric center of each neighboring block.

[0312] When targeting Fig. 9 While the scanning process is performed for the angles B and C shown, the process may be performed jointly or independently. In the example of independent scanning, the previously proposed scanning method may be applied to the angles B and C, respectively. In the example of joint scanning, there may be different methods as follows.

[0313] In one embodiment, a pair scan may be performed. In one example of a pair scan, candidate positions for corners B and C are advanced simultaneously. For ease of illustration and without loss of generality, Fig. 17B As an example. Fig. 17B As shown in , the scanning of corner B starts from the first non-adjacent neighboring block located on the upper side of the current block in the bottom-up direction. The scanning of corner C starts from the first non-adjacent neighboring block located on the left side of the current block in the right-to-left direction. Fig. 17B In the example shown, paired scanning can be defined as the candidate positions of B and C are both advanced by one step unit, where for corner B, one step unit is defined as the height of the current coding block, and for corner C, one step unit is defined as the width of the current coding block.

[0314] In another embodiment, an alternating scan may be performed. In one example of an alternating scan, the candidate positions for corners B and C are alternately advanced. In one step, only the position of B or C may be advanced, while the position of C or B does not change. In one example, the position of corner point B may be gradually increased from the first non-adjacent neighboring block to the distance at the maximum number of non-adjacent neighboring blocks, while the position of corner point C remains at the first non-adjacent neighboring block. In the next round, the position of corner C is moved to the second non-adjacent neighboring block, and the position of corner B is traversed again from the first to the maximum value. The rounds continue until all combinations have been traversed.

[0315] For the purpose of unification, the proposed method of defining scan area and scan distance, scan order and scan termination for deriving inherited merge candidates can be fully or partially reused for deriving constructed merge candidates. In one or more embodiments, the same method defined for inherited merge candidate scans (which includes but is not limited to scan area and scan distance, scan order and scan termination) can be fully reused for constructed merge candidate scans.

[0316] In some embodiments, the same methods defined for inherited merge candidate scans may be partially reused to construct merge candidate scans. Fig.16 An example of this situation is shown. Fig.16 In , each non-adjacent neighboring block has the same block size as the current block, which is similarly defined as the inherited candidate scan, but the whole process is a simplified version because the scan at each distance is limited to only one block.

[0317] Figure 17A-17B represents another example of this situation. Figure 17A-17B In , both non-adjacent inheritance merge candidates and non-adjacent construction merge candidates are defined to have the same block size as the current coding block, while the scanning order, scanning area and scanning termination condition may be defined differently.

[0318] exist Fig.17A In , the maximum distance for the left non-adjacent neighbor is 4 coding blocks, and the maximum distance for the upper non-adjacent neighbor is 5 coding blocks. In addition, at each distance, the scanning direction is bottom-to-top for the left side and right-to-left for the upper side. Fig. 17B In , the maximum distance of non-adjacent neighbors is 4 for both the left and top sides. In addition, scanning at a specific distance is not possible because there is only one block at each distance. Fig.17A In the example, if M qualified candidates are identified, the scanning operation within each distance may be terminated. The value of M may be a predefined fixed value, such as a value of 1 or any other positive integer, or a value determined by the encoder and sent by a signal, or a configurable value at the encoder or decoder. In one example, the value of M may be the same as the size of the merge candidate list.

[0319] exist Figure 17A-17B In the example, if N qualified candidates are identified, the scanning operation at different distances can be terminated. The value of N can be a predefined fixed value, such as a value of 1 or any other positive integer, or a value determined by the encoder and sent by a signal, or a configurable value at the encoder or decoder. In one example, the value of N can be the same as the size of the merge candidate list. In another example, the value of N can be the same as the value of M.

[0320] exist Figure 17A-17B In , non-adjacent spatial neighbors with a closer distance to the current block may be prioritized, which indicates that non-adjacent spatial neighbors with a distance i are scanned or checked before neighbors with a distance i+1, where i may be a non-negative integer representing a specific distance.

[0321] At a certain distance, up to two non-adjacent spatial neighbors are used, which means that at most one neighbor from one side (e.g., left and above) of the current block is selected (if available) for inheriting the candidate derivation or constructing the candidate derivation. Fig.17A As shown in , the order of checking the left neighbor and the upper neighbor is from bottom to top and from right to left respectively. Fig. 17B, this rule can also be applied where the difference can be that at any particular distance, there is only one option for each side of the current block.

[0322] For build candidates, such as Fig. 17B As shown, the positions of a left non-adjacent spatial neighbor and an upper non-adjacent spatial neighbor are first determined independently. After that, the position of the upper left neighbor can be determined accordingly, which can surround the rectangular virtual block together with the left non-adjacent neighbor and the upper non-adjacent neighbor. Then, as Fig. 9 As shown in , the motion information of three non-adjacent neighbors is used to form the CPMV at the upper left (A), upper right (B), and lower left (C) of the virtual block, which is finally projected to the current CU to generate the corresponding construction candidate.

[0323] In step 2, the translation MV at the position of the candidate selected after step 1 is evaluated, and an appropriate affine model can be determined. For easier explanation and without loss of generality, we use Fig. 9 As an example.

[0324] Due to factors such as hardware constraints, implementation complexity and different reference indices, the scanning process may be terminated before a sufficient number of candidates are identified. For example, motion information of the motion field at one or more selected candidates after step 1 may not be available.

[0325] If the motion information of all three candidates is available, the corresponding virtual coding block represents a 6-parameter affine model. If the motion information of one of the three candidates is not available, the corresponding virtual coding block represents a 4-parameter affine model. If the motion information of more than one of the three candidates is not available, the corresponding virtual coding block may not represent a valid affine model.

[0326] In some embodiments, if the upper left corner of the virtually encoded block (e.g., Fig. 9 Motion information is not available at the corner A in the middle, or the upper right corner (e.g. Fig. 9 B) and the lower left corner (e.g. Fig. 9 If motion information at both corners C) in the example above is not available, the virtual block can be set to be invalid and cannot represent a valid model, and steps 3 and 4 can then be skipped for the current iteration.

[0327] In some embodiments, if the upper right corner (e.g. Fig. 9 B) or the lower left corner (e.g. Fig. 9 If angle C) in is not available, but neither is available, then the virtual block can represent a valid 4-parameter affine model.

[0328] In step 3, if the virtual coding block can represent a valid affine model, the same projection process used for the inherited merging candidates can be used.

[0329] In one or more embodiments, the same projection process used for inherited merge candidates may be used. In this case, the 4-parameter model represented by the virtual coding block from step 2 is projected to the 4-parameter model for the current block, and the 6-parameter model represented by the virtual coding block from step 2 is projected to the 6-parameter model for the current block.

[0330] In some embodiments, the affine model represented by the dummy encoded block from step 2 is always projected to a 4-parameter model or a 6-parameter model for the current block.

[0331] Note that according to equations (5) and (6), there may be two types of 4-parameter affine models, where type A is that the upper left CPMV and the upper right CPMV (called V0 and V1) are available, and type B is that the upper left CPMV and the lower left CPMV (called V0 and V2) are available.

[0332] In one or more embodiments, the type of the projected 4-parameter affine model is the same as the type of the 4-parameter affine model represented by the virtual coding block. For example, if the affine model represented by the virtual coding block from step 2 is a 4-parameter affine model of type A or type B, then the projected affine model for the current block is also type A or type B, respectively.

[0333] In some embodiments, the 4-parameter affine model represented by the dummy coding block from step 2 is always projected to the same type of 4-parameter model for the current block. For example, the type A or type B 4-parameter affine model represented by the dummy coding block is always projected to the type A 4-parameter affine model.

[0334] In step 4, based on the projected CPMV after step 3, in one example, the same candidate generation process used in the current VVC or AVS standard may be used. In another embodiment, the temporal motion vector used in the candidate generation process of the current VVC or AVS standard may not be used for the derivation method based on non-adjacent neighboring blocks. When the temporal motion vector is not used, it indicates that the generated combination does not contain any temporal motion vector.

[0335] In step 5, a similarity check may be performed on any newly generated candidate after step 4 relative to all existing candidates already in the merge candidate list. The details of the similarity check have been described in the section on affine merge candidate pruning. If the newly generated candidate is found to be similar to any existing candidate in the candidate list, then the newly generated candidate is removed or pruned.

[0336] In some embodiments, a virtual coding block is formed by determining three corner points A, B, and C, and then the affine model for the virtual coding block is represented by the translation MV of the 4×4 block located at the three corners. Finally, the affine model of the virtual coding block is projected onto the current coding block. This entire process can be used to derive a first type of affine candidate constructed from non-adjacent spatial neighbors (for example, the sub-blocks located by the three corner points A, B, and C are non-adjacent spatial neighbors). In some embodiments, this method can be applied to affine modes, such as affine merge mode and affine AMVP mode, and this method can also be applied to regular modes, such as regular merge mode and regular AMVP mode, because the projected affine model can be used to derive the translation MV based on a specific position (for example, the center position) inside the prediction block or coding block.

[0337] Inheritance-based derivation method for affine construction of merge candidates

[0338] For each affine inheritance candidate, all motion information is inherited from a selected spatial neighboring block encoded in an affine mode. The inherited information includes CPMV, reference index, prediction direction, affine model type, etc. On the other hand, for each affine construction candidate, all motion information is constructed from two or three selected spatial neighboring blocks or temporal neighboring blocks, while the selected neighboring blocks may not be encoded in an affine mode and only require translational motion information from the selected neighboring blocks.

[0339] In this section, a new candidate derivation method is disclosed, which combines the features of inherited candidates and constructed candidates.

[0340] In some embodiments, a combination of inheritance and construction may be achieved by separating the affine model parameters into different groups, where one group of affine parameters is inherited from one neighboring block, and other groups of affine parameters are inherited from other neighboring blocks.

[0341] In one example, the parameters of an affine model may be constructed from two groups. As shown in equation (3), the affine model may contain 6 parameters, including a, b, c, d, e, and f. The translation parameters {a, b} may represent one group, while the non-translation parameters {c, d, e, f} may represent another group. Using this grouping method, the two groups of parameters may be inherited separately from two different neighboring blocks in the first step, and then concatenated / constructed into a complete affine model in the second step. In this case, the group with non-translation parameters must be inherited from an affine-coded neighboring block, while the group with translation parameters may come from any inter-frame coded neighboring block, which may or may not be affine-coded. Note that the proposed scanning method for affine inheritance candidates (e.g., Fig.17AThe method shown in , that is, the scanning method / rule includes the scanning area and distance, scanning order and scanning termination used in the "derivation process based on non-adjacent neighbors for affine inheritance merging candidates" section, and the scanning method can be performed on both adjacent neighboring blocks or non-adjacent neighboring blocks), selects affine coding neighboring blocks from adjacent affine neighboring blocks or non-adjacent affine neighboring blocks. Alternatively, the affine coding neighboring blocks may not actually exist, but are virtually constructed from conventional inter-frame coding neighboring blocks, such as Fig. 17B The method shown in , i.e., the scanning method / rule includes the scanning area and distance, scanning order, and scanning termination used in the "Non-adjacent neighbor-based derivation process for affine constructed merging candidates" section.

[0342] In some examples, the neighboring blocks associated with each group may be determined in different ways. In one approach, the neighboring blocks for different parameter groups may all come from non-adjacent adjacent / neighboring areas, and the scanning method may be designed similarly to the method previously proposed for the derivation process based on non-adjacent neighbors. In another approach, the neighboring blocks for different parameter groups may all come from adjacent adjacent / neighboring areas, and the scanning method may be the same as the current VVC video standard or the AVS video standard. In another approach, the neighboring blocks for different parameter groups may come partially from adjacent areas and partially from non-adjacent adjacent / neighboring areas.

[0343] When scanning adjacent blocks from non-adjacent adjacent / neighboring areas to construct candidates of the current type, the scanning process may be performed differently from the derivation process based on non-adjacent neighbors for affine inheritance candidates. In one or more embodiments, the scanning area, distance, and order may be defined similarly, but the scanning termination rules may be specified differently. For example, non-adjacent adjacent blocks may be scanned exhaustively within the maximum distance defined at each area. In this case, all non-adjacent adjacent blocks within a distance may be scanned by following the scanning order. In some embodiments, the scanning area may be different. For example, in addition to the left area and the upper area, the lower right adjacent and non-adjacent areas of the current coding block may also be scanned to determine neighbors for generating translation parameters and / or non-translation parameters. In addition, the neighbors scanned at the lower right area may be used to find co-located temporal neighbors, rather than spatial neighbors. A scanning criterion may be conditionally based on whether the lower right co-located temporal neighbor has been used to generate an affine constructed neighbor. If it has been used, the scan is not performed, otherwise the scan is performed. Alternatively, if it has been used, this means that the lower right co-located temporal neighbor is available, then the scan is performed, otherwise the scan is not performed.

[0344] When combining several affine parameter groups to construct a new candidate, several criteria may be followed. The first criterion is a qualification criterion. In one example, it may be checked whether one or more associated neighboring blocks for each group use the same reference picture in at least one direction or in both directions. In another example, it may be checked whether one or more associated neighboring blocks for each group use the same precision / resolution on motion vectors.

[0345] When checking a certain criterion, the first X associated neighboring blocks for each group may be used. The value of X may be defined as the same or different values ​​for different parameter groups. For example, the first 1 or 2 neighboring blocks containing non-translation affine parameters may be used, while the first 3 or 4 neighboring blocks containing translation affine parameters may be used.

[0346] The second criterion is to construct a formula. In one example, the CPMV of a new candidate can be derived in the following equation:

[0347]

[0348] where (x,y) is the angular position within the current coding block (e.g., (0,0) for the top-left CPMV, (width,0) for the top-right CPMV), {c,d,e,f} is a set of parameters from one neighboring block, and {a,b} is another set of parameters from another neighboring block.

[0349] In another example, the CPMV of the new candidate may be derived in the following equation:

[0350]

[0351] Where (Δw, Δh) is the distance between the upper left corner of the current coding block and the upper left corner of one of the associated neighboring blocks for a parameter group (e.g., the associated neighboring block in the {a, b} group). The definitions of the other parameters in the equation are the same as in the above example. The parameters can be grouped in another way: (a, b, c, d, e, f) form one group and (Δw, Δh) form another group. And the two parameter groups come from two different neighboring blocks. Alternatively, the value of (Δw, Δh) can be predefined as a fixed value, such as (0, 0) or any constant value that does not depend on the distance between the neighboring block and the current block.

[0352] Fig.18 An example of an inheritance-based derivation method for deriving affine construction candidates is shown. Fig.18 In , there are three steps to derive affine construction candidates. In step 1, according to a specific grouping strategy, the encoder or decoder can perform a scan of adjacent neighboring blocks and non-adjacent neighboring blocks for each group. Fig.18In the case of , two groups are defined, where neighbor 1 is encoded with an affine mode and provides non-translation affine parameters, while neighbor 2 provides translation affine parameters. Figures 15A-15D and Fig.17A The process in the section “derivation process based on non-adjacent neighbors for affine inheritance merge candidates” shown in is used to obtain neighbor 1, and neighbor 1 may be an adjacent neighboring block or a non-adjacent neighboring block of the current block. Fig.16 and Fig. 17B The process shown is used to obtain neighbor 2.

[0353] In some embodiments, neighbor 1 encoded in an affine mode may be scanned out from an adjacent region and / or a non-adjacent region by following the scanning method proposed above. In some embodiments, neighbor 2 encoded in an affine mode or a non-affine mode may also be scanned out from an adjacent region or a non-adjacent region. For example, if the motion information has not been used to derive some affine merge candidates or AMVP candidates, neighbor 2 may come from one of the scanned adjacent regions or non-adjacent regions, or if the co-located TMVP candidate at the lower right position of the current block is available and / or has been used to derive some affine merge candidates or AMVP candidates, neighbor 2 may come from this position. Alternatively, when determining the position of neighbor 2, a small coordinate offset (e.g., +1 or +2 or -1 or -2 for the vertical coordinate and / or the horizontal coordinate) may be applied to provide slightly different motion information for constructing a new candidate.

[0354] In step 2, using the parameters and positions determined in step 1, a specific affine model can be defined that can derive different CPMVs depending on the coordinates (x, y) of the CPMV. Fig.18 As shown, the non-translation parameters {c, d, e, f} can be obtained based on the neighbor 1 obtained in step 1, and the translation parameters {a, b} can be obtained based on the neighbor 2 obtained in step 1. In addition, the current block (x 1 ,y 1 ) and the neighboring block 2(x 2 ,y 2 ) to obtain the distance parameters Δw and Δh. The distance parameters Δw and Δh may indicate the horizontal distance and vertical distance between the current block and neighbor 1 or neighbor 2, respectively. For example, the distance parameters Δw and Δh may indicate the horizontal distance (x 1 -x 2 ) and the vertical distance (y 1 -y 2 ). Specifically, Δw = x 1 -x 2 And Δh=y 1 -y 2 .

[0355] In step 3, two or three CPMVs are derived for the current coding block, which can be constructed to form new affine candidates.

[0356] In some embodiments, other prediction information may be further constructed. If a neighboring block is checked to have the same direction and / or reference picture, the prediction direction (e.g., bidirectional prediction or unidirectional prediction) and index of the reference picture may be the same as the associated neighboring block. Alternatively, the prediction information is determined by reusing the minimum overlap information between associated neighboring blocks from different groups. For example, if the reference index of only one direction from one neighboring block is the same as the reference index of the same direction of another neighboring block, the prediction direction of the new candidate is determined to be unidirectional prediction, and the same reference index and direction are reused.

[0357] In some embodiments, an affine model may be constructed by combining model parameters from different inheritances. In one example, the translation model parameters may be inherited from a translation block (e.g., from adjacent and / or non-adjacent spatially adjacent 4×4 blocks), while the non-translation model parameters may be inherited from an affine coding block (e.g., from adjacent and / or non-adjacent spatially adjacent affine coding blocks). Alternatively, the non-translation model parameters may be inherited from a historically encoded affine block rather than from a non-adjacent spatially adjacent affine coding block scanned out of the display, and the historically encoded affine block may be an adjacent or non-adjacent spatial neighbor. This entire process may be used to derive a second type of affine candidate constructed from non-adjacent spatial neighbors (e.g., the non-translation model parameters may be inherited from non-adjacent spatial neighbors). In some embodiments, this method may be applied to affine modes, such as affine merge mode and affine AMVP mode, and this method may also be applied to conventional modes, such as conventional merge mode and conventional AMVP mode, because the produced affine model can be used to derive a translation MV based on a specific position (e.g., a center position) inside a prediction block or coding block.

[0358] HMVP-based derivation method for affine constructed merge candidates

[0359] has been defined in the current video standards VVC and AVS and in the above sections and Figure 7 In the case of the neighbor-based derivation process described in , a fixed scanning order of neighbor-based blocks is performed to identify two or three neighbor-based blocks. In the case of the non-neighbor-based derivation process, as described in the previous section and Fig. 17B As proposed in , two non-adjacent neighbors are identified during another fixed scanning order. In other words, for both the adjacent neighbor-based derivation method and the non-adjacent neighbor-based derivation method, a certain depth of local scanning inevitably identifies multiple neighbors. This scanning process depends on the local buffer around each current block and also causes a certain amount of computational complexity.

[0360] On the other hand, the HMVP merge mode has been adopted in current VVC and AVS, where the translational motion information from neighboring blocks has been stored in the history table, as described in the introduction. In this case, the scanning process can be replaced by searching the HMVP table.

[0361] Therefore, for the previously proposed non-neighbor-based derivation process and the inheritance-based derivation process, the translational motion information can be obtained from the HMVP table instead of Fig. 17B and Fig.18 The scanning method shown. However, in order to derive affine construction candidates later, position information, width, height and reference information are also required, which can be accessed if the current HMVP table can be modified. Therefore, it is proposed to extend the HMVP table to store additional information in addition to the motion information of each historical neighbor. In one embodiment, the additional information may include the position of affine neighboring blocks or non-affine neighboring blocks, or affine motion information (such as CPMV or equivalent regular motion derived from CPMV (for example, this regular motion may come from an internal sub-block of an affine-encoded neighboring block) reference index, etc.).

[0362] In one or more examples, the non-adjacent neighbor-based derivation process and the inheritance-based derivation process for the affine mode provided above may be replaced by modifying an existing HMVP table.

[0363] In some examples, the translational motion of the derived affine model may be obtained directly from an existing HMVP table, where the translational motion was previously saved when the neighboring block was previously encoded in a conventional inter-frame mode. Alternatively, the translational motion of the derived affine model may still be obtained from an existing HMVP, where the translational motion was previously saved when the neighboring block was previously encoded in an affine mode.

[0364] In some examples, the non-translational motion of the derived affine model is obtained from an existing HMVP table, where the non-translational motion was previously saved when the neighboring block was previously encoded in affine mode. Alternatively, instead of directly saving the reused non-translational motion, the original CPMV of the previously encoded affine neighboring block may be saved. In this case, the existing HMVP table is updated using not only the original CPMV but also the position and size information of the previously encoded affine neighboring block.

[0365] In some other examples, the non-adjacent neighbor-based and inheritance-based derivation processes for affine modes proposed above may be replaced by creating one or more new HMVP tables.

[0366] In some examples, the translational and non-translational motions of the derived affine model may be obtained similarly to the method of modifying an existing HMVP table.

[0367] Candidate derivation methods for affine AMVP and regular merge modes

[0368] As described in the above section, for the affine AMVP mode, an affine candidate list is also required to derive the CPMV predictor. Therefore, all the above-proposed derivation methods can be similarly applied to the affine AMVP mode. The only difference is that when applying the above-proposed derivation methods in AMVP, the selected neighboring block must have the same reference picture index as the current coding block.

[0369] For the normal merge mode, the candidate list is also constructed, but only using the translation candidate MVs and not the CPMVs. In this case, all the derivation methods proposed above can still be applied by adding an additional derivation step. In this additional derivation step, the translation MV for the current block is derived, which can be achieved by selecting a specific pivot position (x, y) within the current block and then following the same equation (3). In other words, to derive the CPMV of the affine block, the three corner positions of the block are used as the pivot position (x, y) in equation (3), while to derive the translation MV of the normal inter-coded block, the center position of the block can be used as the pivot position (x, y) in equation (3). Once the translation MV is derived for the current block, it can be inserted into the candidate list as an additional candidate.

[0370] When new candidates are derived based on the methods proposed above for affine AMVP and regular merge mode, the positions of the new candidates may be reordered.

[0371] In one embodiment, the newly derived candidates may be inserted into the affine AMVP candidate list by following the following sequence:

[0372] (1) Inherit from adjacent spatial neighbors;

[0373] (2) Construction from adjacent spatial neighbors;

[0374] (3) inheritance from non-adjacent spatial neighbors;

[0375] (4) Construction from non-adjacent spatial neighbors;

[0376] (5) translational MVs from adjacent spatial neighbors;

[0377] (6) temporal MVs from adjacent temporal neighbors; and

[0378] (7) Zero MV.

[0379] In another embodiment, the newly derived candidate may be inserted into the affine AMVP candidate list by following the following sequence:

[0380] (1) Inherit from adjacent spatial neighbors;

[0381] (2) Construction from adjacent spatial neighbors;

[0382] (3) inheritance from non-adjacent spatial neighbors;

[0383] (4) translational MVs from adjacent spatial neighbors;

[0384] (5) Construction from non-adjacent spatial neighbors;

[0385] (6) temporal MVs from adjacent temporal neighbors; and

[0386] (7) Zero MV.

[0387] In another embodiment, the newly derived candidate may be inserted into the affine AMVP candidate list by following the following sequence:

[0388] (1) Inherit from adjacent spatial neighbors;

[0389] (2) Construction from adjacent spatial neighbors;

[0390] (3) translational MVs from adjacent spatial neighbors;

[0391] (4) inheritance from non-adjacent spatial neighbors;

[0392] (5) Construction from non-adjacent spatial neighbors;

[0393] (6) temporal MVs from adjacent temporal neighbors; and

[0394] (7) Zero MV.

[0395] In another embodiment, the newly derived candidate may be inserted into the affine AMVP candidate list by following the following sequence:

[0396] (1) Inherit from adjacent spatial neighbors;

[0397] (2) Construction from adjacent spatial neighbors;

[0398] (3) translational MVs from adjacent spatial neighbors;

[0399] (4) temporal MVs from adjacent temporal neighbors;

[0400] (5) inheritance from non-adjacent spatial neighbors;

[0401] (6) Construction from non-adjacent spatial neighbors; and

[0402] (7) Zero MV.

[0403] In another embodiment, the newly derived candidate may be inserted into the affine AMVP candidate list by following the following sequence:

[0404] (1) Inherit from adjacent spatial neighbors;

[0405] (2) Construction from adjacent spatial neighbors;

[0406] (3) translational MVs from adjacent spatial neighbors;

[0407] (4) temporal MVs from adjacent temporal neighbors;

[0408] (5) inheritance from non-adjacent spatial neighbors; and

[0409] (6) Zero MV.

[0410] In another embodiment, the newly derived candidate may be inserted into the affine AMVP candidate list by following the following sequence:

[0411] (1) Inherit from adjacent spatial neighbors;

[0412] (2) Construction from adjacent spatial neighbors;

[0413] (3) translational MVs from adjacent spatial neighbors;

[0414] (4) translational MV from temporal neighbors;

[0415] (5) inheritance from non-adjacent spatial neighbors; and

[0416] (6) Zero MV.

[0417] Note that candidates constructed from non-adjacent spatial neighbors may be referred to as first type candidates and / or second type candidates constructed from non-adjacent spatial neighbors.

[0418] In another embodiment, the newly derived candidate may be inserted into the regular merge candidate list by following the following sequence:

[0419] (1) Spatial MVP from adjacent spatial neighbors;

[0420] (2) temporal MVP from adjacent co-located neighbors;

[0421] (3) spatial MVP from non-adjacent spatial neighbors;

[0422] (4) inherited MVP from non-adjacent spatial affine neighbors;

[0423] (5) constructing MVPs from non-adjacent spatial neighbors;

[0424] (6) History-based MVP from FIFO table;

[0425] (7) Pairwise average MVP; and

[0426] (8)Zero MV.

[0427] Reordering of affine merge candidate lists

[0428] In one embodiment, non-adjacent spatial merge candidates can be inserted into the affine merge candidate list by following the following order: 1. Sub-block based temporal motion vector prediction (SbTMVP) candidate (if available); 2. Inherit from adjacent neighbors; 3. Inherit from non-adjacent neighbors; 4. Construct from adjacent neighbors; 5. Construct from non-adjacent neighbors; 6. Zero MV.

[0429] In another embodiment, non-adjacent spatial merge candidates can be inserted into the affine merge candidate list by following the following order: 1. SbTMVP candidate (if available); 2. Inherit from adjacent neighbors; 3. Construct from adjacent neighbors; 4. Inherit from non-adjacent neighbors; 5. Construct from non-adjacent neighbors; 6. Zero MV.

[0430] In another embodiment, non-adjacent spatial merge candidates can be inserted into the affine merge candidate list by following the following order: 1. SbTMVP candidates (when available); 2. Inherited from adjacent neighbors; 3. Constructed from adjacent neighbors; 4. A set of zero MVs; 5. Inherited from non-adjacent neighbors; 6. Constructed from non-adjacent neighbors; 7. If the list is still not full, the rest are zero MVs.

[0431] In another embodiment, non-adjacent spatial merge candidates can be inserted into the affine merge candidate list by following the following order: 1. SbTMVP candidates (if available); 2. Inherit from adjacent neighbors; 3. Inherit from non-adjacent neighbors with a distance less than X; 4. Construct from adjacent neighbors; 5. Construct from non-adjacent neighbors; 6. Construct from inherited translation neighbors and non-translation neighbors; 7. If the list is still not full, it is zero MV.

[0432] In another embodiment, non-adjacent spatial merge candidates may be inserted into the affine merge candidate list by following the following order: 1. SbTMVP candidate (where available); 2. inherited from adjacent neighbors; 3. inherited from non-adjacent neighbors; 4. first candidate built from adjacent neighbors; 5. first X candidates built from inherited translational and non-translational neighbors; 6. built from non-adjacent neighbors; 7. another Y candidates built from inherited translational and non-translational neighbors; 8. zero MV if the list is still not full.

[0433] In some examples, where the values ​​of X and Y can be predefined fixed values ​​(e.g., value 2), or signaled values ​​received by the decoder (parameters signaled at the sequence / slice / block / CTU level), or configurable values ​​at the encoder / decoder, or values ​​dynamically determined based on the number of available neighbors to the left and above each individual coded block (e.g., X<=3, Y<=3), or any combination of methods for determining the values ​​of X and Y. In one example, the value of X can be the same as the value of Y. In another example, the value of X can be different from the value of Y.

[0434] In another embodiment, non-adjacent spatial merge candidates may be inserted into the affine merge candidate list by following the following order: 1. SbTMVP candidates (where available); 2. Inherited from adjacent neighbors; 3. Inherited from non-adjacent neighbors with a distance less than X; 4. Constructed from adjacent neighbors; 5. Constructed from non-adjacent neighbors with a distance less than Y; 6. Inherited from non-adjacent neighbors with a distance greater than X; 7. Constructed from non-adjacent neighbors with a distance greater than Y; 8. Zero MV. In this embodiment, the value X and the value Y may be predefined fixed values, such as the value 2, or values ​​determined by the encoder and signaled, or configurable values ​​at the encoder or decoder. In one example, the value of X may be the same as the value of Y. In another example, the value of N may be different from the value of M.

[0435] In some embodiments, if a new candidate is derived by using an inheritance-based derivation method that constructs a CPMV by combining affine motion and translation MVs, the position of the new candidate may depend on the positions of other constructed candidates.

[0436] In one embodiment, for different construction candidates, the reordering of the affine merge candidate list may follow the following order:

[0437] (1) Construction from adjacent spatial neighbors;

[0438] (2) constructed by combining adjacent spatial affine neighbors and translating MVs;

[0439] (3) constructed from non-adjacent spatial neighbors; and

[0440] (4) is constructed by combining non-adjacent spatial affine neighbors and translation MVs.

[0441] In another embodiment, for different construction candidates, the reordering of the affine merge candidate list may follow the following order:

[0442] (1) Construction from adjacent spatial neighbors;

[0443] (2) Construction from non-adjacent spatial neighbors;

[0444] (3) constructed by combining adjacent spatial affine neighbors and translating MVs; and

[0445] (4) is constructed by combining non-adjacent spatial affine neighbors and translation MVs.

[0446] In another one or more embodiments, the reordering of the affine merge candidates may partially or completely alternate between candidates of different categories (e.g., the alternation may indicate that candidates from the same category may not be adjacent in the candidate list). In some embodiments, seven categories of affine merge candidates may be placed in the affine merge candidate list:

[0447] (1) SbTMVP candidate (where available);

[0448] (2) Succession candidates from adjacent neighbors;

[0449] (3) Succession candidates from non-adjacent neighbors;

[0450] (4) Construction candidates from adjacent neighbors;

[0451] (5) The second type of constructed candidates from non-adjacent neighbors;

[0452] (7) first type construction candidates from non-adjacent neighbors; and

[0453] (7) Zero MV;

[0454] The specific order discussed above is applicable to any candidate list, including the affine AMVP candidate list, the regular merge candidate list, and the affine merge candidate list.

[0455] Since candidates placed in later positions of the affine merge list may cost higher signaling overhead if selected and signaled by the encoder, the order of the above different categories of candidates can be designed in different ways.

[0456] In one or more embodiments, the order of the candidates may remain the same as the insertion order described above. An adaptive reordering method may then be applied to reorder the candidates; the adaptive reordering may be a template-based method (ARMC) or a non-template-based method, such as a bilateral matching-based method.

[0457] In one or more embodiments, the order of candidates may be reordered in a specific pattern. The specific pattern may be applied to any candidate list, including an affine AMVP candidate list, a regular merge candidate list, and an affine merge candidate list.

[0458] In some embodiments, the reordering mode may depend on the number of available candidates for each category.

[0459] In one example, the reordering mode may be defined as follows:

[0460] (1) SbTMVP candidate (where available);

[0461] (2) Succession candidates from adjacent neighbors;

[0462] (3) Construction candidates from adjacent neighbors;

[0463] (4) the top X successor candidates from non-adjacent neighbors (e.g., X may be a pre-fixed number such as 1 or a signaled number);

[0464] (5) the first Y construction candidates of the second type of construction candidates from non-adjacent neighbors (e.g., Y can be defined similarly to X);

[0465] (6) the first Z construction candidates of the first type of construction candidates from non-adjacent neighbors (e.g., Z can be defined similarly to X);

[0466] (7) remaining successor candidates from non-adjacent neighbors (e.g., X may be a pre-fixed number such as 1 or a signaled number);

[0467] (8) the remaining construction candidates of the second type of construction candidates from non-adjacent neighbors (e.g., Y can be defined similarly to X);

[0468] (9) the remaining construction candidates of the first type from non-adjacent neighbors (e.g., Z may be defined similarly to X); and

[0469] (10)Zero MV.

[0470] In one or more embodiments, the reordering mode may be an alternating method that may merge different candidates from different categories. In one example, the alternating mode may be defined as follows:

[0471] (1) SbTMVP candidate (where available);

[0472] (2) Succession candidates from adjacent neighbors;

[0473] (3) the top X1 construction candidates from adjacent neighbors;

[0474] (4) the first Y1 construction candidates of the second type of construction candidates from non-adjacent neighbors;

[0475] (5) the first Z1 successor candidates from non-adjacent neighbors;

[0476] (6) the first K1 construction candidates of the first type of construction candidates from non-adjacent neighbors;

[0477] (7) the top X2 construction candidates from adjacent neighbors;

[0478] (8) the first Y2 construction candidates of the second type of construction candidates from non-adjacent neighbors;

[0479] (9) the first Z2 successor candidates from non-adjacent neighbors;

[0480] (10) the first K2 construction candidates of the first type of construction candidates from non-adjacent neighbors;

[0481] (11)…

[0482] (12) The first Xi construction candidates from the adjacent neighbors;

[0483] (13) the previous construction candidate of the second type of construction candidate from non-adjacent neighbors;

[0484] (14) The first successor candidates from non-adjacent neighbors;

[0485] (15) the first Ki construction candidates of the first type of construction candidates from non-adjacent neighbors; and

[0486] (16)Zero MV.

[0487] The value of (Xi, Yi, Zi, Ki) can be a pre-fixed number (such as 1) or a signaled number. If the number of available candidates for one category is less than the number of available candidates for other categories, the position of the candidate for that category is skipped and the remaining available candidates for other categories will occupy that position.

[0488] In one or more embodiments, the reordering mode may be a combined version that takes into account both availability and alternation methods. In one example, the combined mode may be defined as follows:

[0489] (1) SbTMVP candidate (where available);

[0490] (2) Succession candidates from adjacent neighbors;

[0491] (3) the top X1 construction candidates from adjacent neighbors;

[0492] (4) all available construction candidates of the second type from non-adjacent neighbors;

[0493] (5) remaining construction candidates from adjacent neighbors;

[0494] (6) the first Z1 successor candidates from non-adjacent neighbors;

[0495] (7) the first K1 construction candidates of the first type of construction candidates from non-adjacent neighbors;

[0496] (8)…

[0497] (9) The first successor candidates from non-adjacent neighbors;

[0498] (10) the first Ki construction candidates of the first type of construction candidates from non-adjacent neighbors; and

[0499] (11)Zero MV.

[0500] In some examples, the above reordering methods may be selected and applied according to different factors.

[0501] In one or more examples, the reordering method may be selected based on the type of video frame / strip. For example, for a low-latency picture or strip, all candidates of the first type of construction candidates from non-adjacent neighbors may be placed after all construction candidates from adjacent neighbors. For a non-low-latency picture or strip, the first K1 candidates of the first type of construction candidates from non-adjacent neighbors may be placed after the first K2 construction candidates from adjacent neighbors, and the remaining candidates of the first type of construction candidates from non-adjacent neighbors may be placed after the remaining construction candidates from adjacent neighbors.

[0502] Improved reordering of affine candidate lists

[0503] Based on the candidate derivation method proposed above, one or more candidates may be derived for an existing affine merge candidate list or an affine AMVP candidate list or a regular merge candidate list, where the size of the corresponding list may be adjusted statically (e.g., configurable size) or adaptively (e.g., dynamically changed according to availability at the encoder and then sent to the decoder with a signal). Note that when one or more new candidates are derived for a regular merge candidate list, the new candidates are first derived as affine candidates and then converted into translational motion vectors by using a pivot position (e.g., a center sample or pixel position) within the coding block and an associated affine model before being inserted into the regular merge candidate list.

[0504] In one or more embodiments, after updating or constructing the candidate list by adding some new candidates derived by the candidate derivation method proposed above, an adaptive reordering method such as ARMC may be applied to one or more of the above candidate lists.

[0505] In another embodiment, a temporal candidate list may be created first, wherein the temporal candidate list may have a size larger than an existing candidate list (e.g., an affine merge candidate list, an affine AMVP candidate list, a regular merge candidate list). Once the temporal candidate list is constructed by adding newly derived candidates and statically sorted using the insertion method proposed above, an adaptive reordering method such as ARMC may be applied to reorder the temporal candidate list. After the adaptive reordering, the first N candidates of the temporal candidate list are inserted into the existing candidate list, where the value of N may be a fixed value or a configurable value. In one example, the value of N may be the same as the size of the existing candidate list, where the N candidates selected from the temporal candidate list are located.

[0506] In the above application scenarios where an adaptive reordering method such as ARMC is applied, the following methods may be used to improve performance and / or reduce the complexity of the applied reordering method.

[0507] In some embodiments, when the template matching cost is used to reorder different candidates, a cost function such as the sum of absolute differences (SAD) between the samples of the template of the current block and its corresponding reference samples can be used. The reference samples of the template can be located by the same motion information of the current block. In the case where fractional motion information is used for the current block, an interpolation filtering process can be used to generate prediction samples of the template. Since the generated prediction samples are only used to compare the motion accuracy between different candidates, rather than for final block reconstruction, the prediction accuracy of the template samples can be relaxed by using an interpolation filter with a smaller tap. For example, in the case of adaptively reordering the affine merge candidate list, a 2-tap or 4-tap or any other shorter length (e.g., 6-tap, 8-tap) interpolation filter can be used to generate prediction samples for the selected template of the current block. Or even the closest integer sample (skipping the interpolation filtering process completely) can be used as the prediction sample of the template. When the template matching method is used to adaptively reorder candidates in other candidate lists (e.g., a conventional merge candidate list or an affine AMVP candidate list), an interpolation filter with a smaller tap can be similarly used.

[0508] In some embodiments, when the template matching cost is used to reorder different candidates, a cost function such as the SAD between the samples of the template of the current block and its corresponding reference samples can be used. The corresponding reference samples can be located at integer positions or fractional positions. When locating fractional positions, a certain level of prediction accuracy can be achieved by performing an interpolation filtering process. Due to limited prediction accuracy, the matching costs calculated for different candidates may contain noise level differences. In order to reduce the impact of noise level cost differences, the calculated matching costs can be adjusted by removing a few of the least significant bits before the candidate sorting process.

[0509] In some embodiments, if enough candidates cannot be derived by using different derivation methods, the candidate lists can be padded with zero MVs at the end of each list. In this case, the candidate cost can be calculated only for the first zero MV, and the remaining zero MVs can be statically assigned arbitrarily large cost values ​​so that these repeated zero MVs are placed at the end of the corresponding candidate lists.

[0510] In some embodiments, all zero MVs may be statically assigned an arbitrarily large cost value such that all zero MVs are placed at the end of the corresponding candidate list.

[0511] In some embodiments, early termination methods may be applied to the reordering method to reduce the complexity on the decoder side.

[0512] In one or more embodiments, when building a candidate list, different types of candidates may be derived and inserted into the list. If one candidate or one type of candidate does not participate in the reordering process, but is selected and signaled to the decoder, the reordering process applied to other candidates may be terminated early. In one example, in the case of applying ARMC to an affine merge candidate list, the SbTMVP candidate may be removed from the reordering process. In this case, if the signaled merge index value for an affine coded block indicates an SbTMVP candidate at the decoder side, the ARMC process may be skipped or terminated early for this affine block.

[0513] In another embodiment, if a candidate or a type of candidate does not participate in the reordering process, is not selected and sent to the decoder with a signal, both the derivation process and the reordering process for the specific candidate or the specific type of candidate can be skipped. Note that the skipped derivation process and reordering process are only applied to the specific candidate or the specific type of candidate, while the remaining candidates or candidate types are still subjected to these two processes, wherein skipping the derivation process indicates skipping the relevant operations of deriving the specific candidate or the specific type of candidate, but the predefined list position of the specific candidate or the specific type of candidate can still be maintained (for example, according to a predefined insertion order), and only the candidate content such as motion information may be invalid due to the skipped derivation process. Similarly, during the reordering process, the cost calculation of the specific candidate or the specific type of candidate can be skipped, and after reordering other candidates, the list position of the specific candidate or the specific type of candidate may not be changed.

[0514] Motion information storage

[0515] When scanning non-adjacent spatial neighbors based on the candidate derivation method proposed above, the selected non-adjacent spatial neighbors can be affine coded blocks or non-affine coded blocks (e.g., conventional inter-frame AMVP or merged coded blocks). In the case of non-affine coded blocks, the motion information may include the translation MV and the corresponding reference index at each direction. In the case of affine coded blocks, the motion information may include the CPMV and the corresponding reference index at each direction as well as the position and size of the affine coded blocks.

[0516] For affine coded blocks or non-affine coded blocks, once these blocks have been encoded, it may be necessary to save the motion information of these blocks in memory. In order to save memory usage, non-adjacent spatial neighbors can be restricted to a specific area.

[0517] like Fig.21 As shown, the allowed non-adjacent region for scanning non-adjacent spatially neighboring blocks may be restricted to a finite region size.

[0518] In one or more embodiments, the restricted region may be applied to either affine space neighboring blocks or non-affine space neighboring blocks.

[0519] The size of the allowed non-adjacent region may be defined according to the size of the current CTU (eg, an integer (eg, 1 or 2 or other integers) or a fraction (eg, 0.5 or 0.25 or other fractions) of the current CTU size).

[0520] The size of the allowed non-contiguous region may be defined according to a fixed number of pixels or samples (eg, 128 samples above the current CTU and / or to the left of the current CTU).

[0521] The size (eg, in terms of CTU size or number of samples) may be a pre-fixed value or a signaled value determined at the encoder and carried in the bitstream.

[0522] In some other examples, the size of the restricted area may be defined separately for the top non-adjacent neighboring block and the left non-adjacent neighboring block.

[0523] In one example, the upper non-adjacent neighboring blocks may be limited to being within the current CTU, or outside the current CTU but within a fixed number of samples / pixels from the top of the current CTU, so that no additional line buffer is required to save the motion information of the upper non-adjacent neighboring blocks. For example, if the adjacent / neighboring area of ​​8 sample rows from the top of the current CTU is already covered by the existing line buffer, the fixed number may be defined as 8.

[0524] In another example, the left non-adjacent neighboring blocks may be restricted to be within the current CTU, or outside the current CTU but within a predefined or signaled number of samples / pixels from the left boundary of the current CTU.

[0525] In some examples, such as Figure 27A-27B As shown in , if the allowed non-adjacent area above the current CU (for non-adjacent affine neighbors or non-affine neighbors) is outside the current CTU, the allowed non-adjacent area may have a large memory cost. In this case, the actual memory cost increases proportionally with the picture width and the maximum allowable scan distance in the vertical direction. In order to reduce the line buffer (i.e., the upper non-adjacent area outside the current CTU) cost, the height of the upper non-adjacent area outside the current CTU can be limited to a value h, such as Fig.27A In some examples, this value h may be configurable or signaled to the decoder. In the case where affine motion and non-affine motion are stored in separate buffers, for Fig.27B With reference to the example shown in , there are different ways to save the motion in the line buffer, as described below.

[0526] In one example, the line buffer for storing affine motion may indicate that a buffer region where CU B is located is set to invalid because CU B is not an affine CU.

[0527] In another example, the line buffer for storing affine motion may indicate that the buffer region where CU B is located is set to valid and the affine motion is copied from CU A because CU A is an adjacent affine neighbor of CU B.

[0528] exist Figure 27A-27B In order to make it easier to implement, the height value h and the width value w may be set to a multiple of 4. In one example, the value h and the value w may be set to the minimum value (e.g., 4) of the non-affine CU. When the allowed non-adjacent area for scanning non-adjacent spatial neighboring blocks is limited to a limited area size, such as Fig.21 and Figure 27A-27B In the example in , the scanned non-adjacent neighbor position may be outside the allowed non-adjacent area. In this case, different methods can be used to solve the problem, as described below.

[0529] In one example, the scanning process may indicate that this scan location does not have valid neighbor information.

[0530] In another example, the scanning process may project or clip this out-of-range location to another location within the allowed non-adjacent area. Fig.28As shown, there are two positions 2801 (i.e., two dotted points 2801) outside the range of the allowable non-adjacent area. The two positions 2801 are projected / cropped to two other positions 2802, which are respectively at the same vertical / horizontal coordinates but within the allowable non-adjacent area. The projected / cropped new position 2802 may be located on the boundary of the allowable non-adjacent area closest to the original position. In the case where the values ​​w and h of the allowed non-adjacent area outside the current CTU are set to the minimum value of the non-affine CU (e.g., 4), the projected / cropped new position 2802 may be interchangeably set on one boundary or the other because the buffer is too small to store motion information from only one CU, and in this case, there is no difference in cropping to one boundary side or the other.

[0531] In some other examples, the allowed non-adjacent spatial regions may include three regions. Fig.29 As shown, the three regions are the upper left region, the left region, and the upper region outside the current CTU and the spatial region adjacent to the current CTU. The height of the upper left region and the upper region in the allowed non-adjacent spatial region is defined as h, and the width may depend on the picture width. The width of the allowed non-adjacent spatial region on the left is defined as w, and the height is equal to the height of the current CTU. Fig.29 As shown, if the scanned non-adjacent position 2906-1 / 2906-2 / 2906-3 is outside the allowed spatial area, the new position 2907-1 / 2907-2 / 2907-3 after projection / cropping can be defined in different ways. For the upper left non-adjacent area, the new position after projection / cropping is always the pixel position adjacent to the upper left position of the current CTU. For example, if the upper left position of the current CTU is (ctu_x, ctu_y), the new position after projection / cropping is (ctu_x-1, ctu_y-1). For the upper non-adjacent area, the new position after projection / cropping has the same horizontal coordinate, but the vertical coordinate becomes (ctu_y-1). For the left non-adjacent area, the new position after projection / cropping has the same vertical coordinate, but the horizontal coordinate becomes (ctu_x-1).

[0532] When the motion information of the affine coding block is stored in the memory, the motion information including CPMV, reference index, block size and position can be stored at the granularity of the minimum affine block size (e.g., 8×8 block). In the case where the current affine coding block is a coding unit with a size larger than the minimum affine block, the motion information can be stored in different methods.

[0533] In one or more embodiments, the motion information stored at each minimum affine block (e.g., 8×8 block) within the current block is simply a duplicate copy of the motion information of the current block. Figures 22A to 22B The position and size of each minimum affine block (called the parent block in Figures 22A to 22B An example of this is shown in Fig.22A , where the current block (called the parent block) has a size of 24×16 and the smallest affine block (called the child block) has a fixed size of 8×8.

[0534] In one or more other embodiments, in each minimum affine block (in Figure 22A-22B The motion information saved at the minimum affine block (called a sub-block in the example) is the motion information that has been projected to this minimum affine block. Since the position of each minimum affine block is known (the upper left corner of each minimum affine block), and the size of each minimum affine block is also known (the minimum size is 8×8), there is no need to save the current block (in the example). Figure 22A-22B The position and size information of the parent block. Fig. 22B An example of this is shown in .

[0535] When using Fig. 22B When using the storage method of , the conversion from affine motion to regular / translational motion can also be simplified as follows.

[0536] In some examples, assuming the size of the smallest non-affine block is 4×4, for each 8×8 affine block, the regular / translational motion at each inner non-affine block may be calculated as follows.

[0537] by Fig.23 As an illustrative example. For this minimal affine block, there is Fig. 22B Based on the affine model shown in equation (2), the regular / translational motion can be derived as follows (e.g., ignoring some precision-dependent shifts). The example provided below is based on Fig.23 The center point of each 4×4 block shown in , but the present disclosure is not limited to using the center point to derive the translation MV for each block.

[0538] In some examples, for the top left sub-block B1, MV1_x=e+(a>>2)+(c>>2), and MV1_y=f+(b>>2)+(d>>2), where a=CPMV2_x–CPMV1_x, b=CPMV2_y–CPMV1_y, c=CPMV3_x–CPMV1_x, d=CPMV3_y–CPMV1_y, e=CPMV1_x, and f=CPMV1_y.

[0539] In some examples, for the top right sub-block B2, MV2_x=MV1_x+(a>>1), and MV2_y=MV1_y+(b>>1).

[0540] In some examples, for the bottom left sub-block B3, MV3_x=MV1_x+(c>>1), and MV3_y=MV1_y+(d>>1).

[0541] In some examples, for the bottom right sub-block B4, MV4_x=MV1_x+((a+c)>>1), and MV4_y=MV1_y+((b+d)>>1).

[0542] Alternatively or additionally, the motion information of the affine encoded block can be stored at a different granularity a×b (e.g., 16×16 or 16×32 or 32×16 or 32×32 granularity, etc.) instead of the minimum affine block size (e.g., 8×8 granularity), where the granularity values ​​of a and b can be configurable, or determined at the encoder and then signaled to the decoder. Without loss of generality, it takes a granularity of 16×16 (e.g., a=b=16) as an illustrative example. If it is further assumed that the minimum affine block size is 8×8, this indicates that each 16×16 block can only store one affine motion information set, which includes two or three CPMVs and represents a single affine model, even though the four 8×8 sub-blocks within this 16×16 block may come from more than one affine block, such as Fig.25 As shown in Fig.25 In , four 8×8 sub-blocks A, B, C and D form a 16×16 block / region, and only one affine model information is saved. However, the four 8×8 sub-blocks come from four different affine blocks, which means four affine models and include four affine motion information sets. In this case, there may be different ways to derive and save a single affine information set.

[0543] In one or more examples, one of a plurality of available affine motion information sets may be selected and saved. In one example, affine motion information at a fixed or configurable position (e.g., the top left minimum affine block) is selected for motion storage. In another example, average affine motion information of a plurality of models may be calculated for motion storage.

[0544] In some examples, affine motion information at selected neighboring affine blocks may be simplified / compressed prior to storage.

[0545] In one example, it is proposed that the selected neighboring affine blocks are always 4-parameter models and only two CPMVs are saved.

[0546] In another example, it is proposed that the selected neighboring affine blocks are always unidirectionally predicted and only one direction of affine motion is saved.

[0547] In another example, instead of directly saving the CPMV, it is proposed to save the affine model parameters converted from the corresponding CPMV, so that there is no need to save the size information (e.g., width and height) of the neighboring blocks. In this case, it may still be necessary to save the upper left CPMV to provide translation motion.

[0548] In another example, each saved CPMV can be compressed before storage to further reduce the memory size. One example is to use general techniques for data compression. For example, saving a composite value from an exponent and mantissa to approximately represent each saved CPMV is provided.

[0549] In another method, the affine motion information of non-adjacent affine coded blocks can be obtained by deriving it from existing stored translational motion information. An example of this method is Figure 30A-Figure 30C Shown in.

[0550] In such Fig. 30A In the first step shown, each scanned non-adjacent position is normalized / projected / cropped to the upper left position of the predefined / configured sub-block where the scanned non-adjacent position is located. The size of the sub-block depends on the storage granularity, such as Fig. 30A The positions of point 3001 and point 3002 are evenly distributed in the CTU, wherein point 3001 indicates that there is no valid affine model information at this position, and point 3002 indicates that there is valid affine model information at this position.

[0551] In such Fig. 30B In the second step shown in , information at each normalized / projected / cropped position is obtained to determine whether valid affine model information is available. Fig. 30B As shown, the three pieces of information include the position, width and height of the non-adjacent affine CU. If both the width and height are positive values, it indicates that valid affine information is available, otherwise it is not available.

[0552] The position information including the horizontal coordinate and the vertical coordinate may be stored in different ways. In one example, the absolute position value of each non-adjacent affine CU may be saved. In some examples, the position of each affine CU may be at least a multiple of the smallest CU, so that the position value may be right-shifted by 2 bits (e.g., when the smallest CU size is 4×4) or 3 bits (e.g., when the smallest CU size is 8×8) before storage to reduce memory cost.

[0553] In another example, the relative position value of each non-adjacent affine CU may be saved. In some examples, the CTU position information where each non-adjacent affine CU is located is known. In this case, for each non-adjacent affine CU, the position of the non-adjacent affine CU relative to the upper left pixel position of its associated CTU may be saved. For example, if the maximum CTU size is 256×256 and the minimum CU size is 4×4, the dynamic range of the position information may be represented as 4×[0,63], where values ​​in the range [0,63] may be represented by 6 bits.

[0554] The size information including width and height may be stored in different ways. In one method, the size may be stored as an actual value, and the dynamic range is (0, maximum CTU size]. In another example, the size information may be stored as a lookup table, where the table only includes a set of all possible size values, including 8, 16, 32, 64, 128 up to the maximum CTU size. In this case, 3 bits are sufficient to represent the width value or the height value. In another example, if the size information is always a power of 2, a right shift operation may be applied to the size information storage (e.g., right shift 3 bits before storing the size information).

[0555] In such Fig. 30C In the third step shown, if valid affine information is available, three sub-blocks (eg, Fig. 30C In some examples, the three sub-blocks are top-left, top-right, and bottom-left intra sub-blocks of the associated affine CU.

[0556] In one example, if the position 3004 of the non-adjacent affine CU represented is (x, y), the size of this affine CU is w and h, and the size of each intra-frame sub-block is 4×4, then the center position of the upper left sub-block can be derived as (x+2, y+2), the center position of the upper right sub-block can be derived as (x+w-2, y+2), and the center position of the lower left sub-block can be derived as (x+2, y+h-2).

[0557] like Fig. 30CAs shown, since the pivot position used to derive the translation / regular motion information of each sub-block is different from the pixel position for deriving the original CPMV, there may be different methods for deriving the affine model information of non-adjacent affine CUs. In one example, the translation / regular motion information of the three sub-blocks that are already available in the existing motion storage can be directly used as the CPMV for the selected non-adjacent affine CU to derive the affine model parameters. In another example, the translation / regular motion information of the three sub-blocks can be directly used as the CPMV, but the size and position of the non-adjacent affine CU can be modified to match the position difference between the sub-block MV and the original CPMV. In one or more examples, the size is reduced from the original width × height to (width-sub-block width) × (height-sub-block height), and the position is changed from the original (x, y) to (x + half of the sub-block width, y + half of the sub-block height).

[0558] In another approach, the affine motion information of non-adjacent affine coded blocks can still be obtained by deriving from the existing stored translational motion information, but the required information and memory size can be further reduced. An example of this is in Figure 31A-Figure 31C Shown in.

[0559] Fig.31A The examples in Fig. 30A This shows that the storage granularity in this method can be Figure 30A-Figure 30C The methods in are defined similarly.

[0560] exist Fig.31B In , only one piece of information is stored, namely the affine CU position. This shows that this method requires lower memory storage. Fig. 30B The required position information is stored similarly, where an absolute position value or a relative position value (e.g., relative to the upper left corner of the current CTU) may be stored. In order to indicate whether the stored position information is valid, a variety of methods may be proposed. In one method, a specially selected position may be used to indicate invalid position information. For example, the lower right corner position of the current CTU (when a relative position value is stored) or the current picture / slice (when an absolute position value is stored) may be used to distinguish valid position information from invalid position information.

[0561] exist Fig. 31C In , for affine model export, the internal sub-blocks are selected differently. Note that due to Fig.31BNo width and height information is stored in , so the upper right internal sub-block and the lower left internal sub-block within the current affine CU cannot be determined. According to the actual definition of the minimum width and the minimum height, the selection of the internal sub-block can be adaptively determined. In one or more embodiments, in some traditional video codecs, such as VVC / AVS / ECM, the minimum width and the minimum height of the affine CU may always be defined as a value of 8, and the width and height of any internal sub-block within the affine CU may be constantly defined as a value of 4. In this case, there are at least four sub-blocks in each affine CU, and three sub-blocks 3103 may be selected as: the upper left sub-block (which can be obtained due to the affine CU position information), the sub-block to the right of the upper left sub-block, and the sub-block immediately below the upper left sub-block. The position 3104 of the non-adjacent affine CU represented by is (x, y). Once these three sub-blocks are selected, the remaining process details for affine model derivation may be similarly defined as Fig. 30C Note that if Fig. 31C If these three sub-blocks are selected as shown, the width and height of the associated affine CU may be set to the minimum width value and the minimum height value, since the actual width value and the height value are not stored and cannot be retrieved.

[0562] In the above examples, the methods may be applied to motion information storage in any combination. For example, a defined restricted area for non-adjacent neighboring blocks may be combined with the use of compressed affine motion information.

[0563] MMVD-based candidate derivation method for affine AMVP and merge mode

[0564] In MMVD mode, the best MVD information is selected at the encoder side based on the rate-distortion optimization (RDO) method and then signaled to the decoder side. The trade-off between signaling cost and potential coding bit savings determines whether MMVD mode is applied to a coding block.

[0565] In some examples, using MVD information to refine existing candidates in the affine AMVP and / or affine merge candidate list is provided. Then, the refined new candidate is inserted into the existing affine AMVP and / or affine merge candidate list.

[0566] In some examples, the number of available combinations for MVD information, such as motion magnitude (e.g., offset value) and motion direction (e.g., symbol value) may be the same or different than the existing MMVD mode in VVC. In one example, a smaller number of offset values, such as {1, 2, 4, 8, 16}, may be used. In another example, a different or the same set of direction values ​​as the existing MMVD mode may be used.

[0567] When selecting one or more base MVs, in some examples, the base MV is any one of the candidates from the existing affine AMVP and / or merge candidate list, and there may be multiple ways to determine the selection of potential base MVs. In one or more examples, the base MV may be selected from the candidate list before or after an adaptive reordering method such as ARMC is applied to the candidate list.

[0568] In one or more examples, a single base MV or multiple base MVs may be selected from the candidate list. For example, when a single base MV is selected, one or more combinations (e.g., Y combinations) of MVD information may be selected to refine the single base MV, indicating that Y new candidates may be generated (e.g., each combination of MVD information is applied to the base MV to generate one new candidate) and inserted into the candidate list. For another example, when multiple base MVs (e.g., X base MVs) are selected, one or more combinations (e.g., Y combinations) of MVD information may be selected to refine each selected base MV, indicating that X times Y new candidates may be generated and inserted into the candidate list.

[0569] When one or more basic MVs are selected, the index of each selected basic MV may be determined in different ways. In one or more examples, if the affine MMVD mode is enabled for the current encoding and decoding process, the index of the selected basic MV may be determined by avoiding the basic MVs that have been selected in the MMVD mode.

[0570] In one or more examples, the index of the selected basic MV can be determined by following a predefined order. For example, N candidates are selected as basic MVs in sequence from the beginning of the list. If any basic MV has been selected by the current affine MMVD mode, this basic MV can be skipped.

[0571] Newly generated candidates after different combinations of MVD refinement can be directly inserted into the existing candidate list. Alternatively, another round of re-ranking process can be applied to all new candidates, and the top Z candidates with smaller matching costs (e.g., template matching costs or bilateral matching costs) can be selected to be inserted into the candidate list.

[0572] Improved regression-based affine candidate derivation

[0573] In the regression-based affine candidate derivation method, there are several steps performed by the encoder / decoder, as follows:

[0574] Step 1: Scan and determine the selected neighboring affine CUs.

[0575] Step 2: Use only the motion information from the internal sub-blocks of the selected neighboring affine CU as input and solve the linear regression model to derive the parameters of an affine model

[0576] Step 3: Use the neighboring (adjacent or non-adjacent) affine CU internal motion information and adjacent 4×4 sub-block motion information of the current CU ( Fig.32 The upper and left gray sub-blocks shown in ) are taken as input and a linear regression model is solved to derive the parameters of another affine model

[0577] Step 4: Convert the derived affine model parameters into an affine CPMV set, and insert the CPMV into the existing affine merge candidate list and AMVP candidate list.

[0578] Note that in the above steps 2 and 3, the difference is whether the sub-block motion information only from the affine CU or the motion information of the adjacent 4×4 sub-block is used as the input of the linear regression calculation. In addition, for each selected adjacent affine CU, two affine models can be derived: one derived only from the adjacent affine sub-block, and the other derived from both the adjacent affine sub-block and the adjacent 4×4 sub-block.

[0579] In some examples, several methods are proposed to improve the regression method.

[0580] For the scanning and determination of neighboring affine CUs, the above-mentioned methods can be used in any combination. In one embodiment, Fig.17A In order to constrain memory usage, when the selected non-adjacent affine CU is located inside or outside the current CTU, Fig.28 or Fig.29 The method proposed in can be used to distinguish the acquisition of non-adjacent adjacent affine CU motion information. When the selected non-adjacent affine CU is located within the current CTU, the associated motion information can be acquired with different granularity and availability, and the method proposed in Figure 30 or Figure 31 can be applied as an alternative solution.

[0581] As explained above, for each selected neighboring affine CU, two affine models can be derived. For the first type of affine model, only the motion information of the internal sub-blocks of the selected neighboring affine CU is used as the input of the linear regression calculation. Since the affine model currently used in the main traditional video codecs such as VVC / AVS is actually a linear model, all internal sub-blocks basically follow the same linear model with only rounding changes. In this case, it is proposed to replace the linear regression calculation used for the first type of affine model derivation with a simple affine model inheritance, and the inheritance process is the same as the projection method used in the current main traditional video codecs or the method proposed above in the part of the non-adjacent neighbor-based derivation process for affine inheritance merge candidates. For the case of memory constraints, it can be achieved by following Fig. 30C or Fig. 31CThe method proposed in obtains the affine motion before the inherited projection.

[0582] For the second type of affine model, both the motion information of the internal sub-block of the selected neighboring affine CU and the motion information of the neighboring 4×4 sub-block are used as inputs for the linear regression calculation. It is proposed to control the ratio of motion information from the affine internal sub-block and from the neighboring 4×4 sub-block. In particular, when using a 4×4 sub-block such as Fig. 31C In case of memory constraints, the number of available internal affine sub-blocks may be reduced (e.g., only four internal sub-blocks may be available for the affine CU). In one or more embodiments, when the number of available affine sub-blocks is much smaller than the number of available adjacent 4×4 sub-blocks (e.g., a threshold ratio value may be defined as th1), the derivation process may be skipped or terminated early. In another one or more embodiments, when the number of available affine sub-blocks is much larger than the number of available adjacent 4×4 sub-blocks (e.g., a threshold ratio value may be defined as th2), the derivation process may be skipped or terminated early. It should be noted that the threshold ratio value may be predefined as a constant value, or may be directly signaled to the decoder or may be signaled to a lookup table index of the decoder.

[0583] Fig.24 A computing environment (or computing device) 2410 coupled to a user interface 2460 is shown. The computing environment 2410 may be part of a data processing server. In some embodiments, the computing device 2410 may perform any of the various methods or processes (such as encoding / decoding methods or processes) described above according to various examples of the present disclosure. The computing environment 2410 may include a processor 2420, a memory 2440, and an I / O interface 2450.

[0584] The processor 2420 generally controls the overall operation of the computing environment 2410, such as operations associated with display, data acquisition, data communication, and image processing. The processor 2420 may include one or more processors to execute instructions to perform all or some steps in the above method. In addition, the processor 2420 may include one or more modules that facilitate interaction between the processor 2420 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a GPU, etc.

[0585] The memory 2440 is configured to store various types of data to support the operation of the computing environment 2410. The memory 2440 may include predetermined software 2442. Examples of such data include instructions for any application or method operating on the computing environment 2410, video data sets, image data, etc. The memory 2440 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0586] I / O interface 2450 provides an interface between processor 2420 and peripheral interface modules such as a keyboard, click wheel, buttons, etc. Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 2450 may be coupled to an encoder and a decoder.

[0587] In some embodiments, a non-transitory computer-readable storage medium is also provided, which includes a plurality of programs, for example, included in the memory 2440, and can be executed by the processor 2420 in the computing environment 2410 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, RAM, CD-ROM, tape, floppy disk, optical data storage device, etc.

[0588] A non-transitory computer-readable storage medium has stored therein a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, causes the computing device to perform the above-described method for motion prediction.

[0589] In some embodiments, the computing environment 2410 may be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), graphic processing units (GPUs), controllers, microcontrollers, microprocessors, or other electronic components for executing the above methods.

[0590] Fig.33 is a flowchart illustrating a method for video decoding according to an example of the present disclosure.

[0591] In step 3301, the processor 2420 on the decoder side may obtain a normalized position for an affine CU at a non-adjacent position based on a storage granularity, wherein the non-adjacent position is obtained by scanning an area that is not adjacent to the current CU.

[0592] like Fig.31A As shown, each scanned non-adjacent position may be normalized / projected / cropped to the upper left position of the predefined / configured sub-block where the scanned non-adjacent position is located, for example, normalized / projected / cropped to a point close to 3101 or 3102 .

[0593] In step 3302, the processor 2420 at the decoder side may determine whether valid affine model information is stored for the normalized position, wherein the valid affine model information includes the position of the affine CU. Fig.31B As shown, only one piece of information is stored, namely the affine CU position, thus requiring lower memory storage.

[0594] In step 3303 , the processor 2420 at the decoder side may, in response to determining that valid affine model information is stored for the normalized position, derive affine model information for the affine CU based on the valid affine model information stored for the normalized position.

[0595] In some examples, the processor 2420 on the decoder side may determine that valid affine model information is stored for the normalized position in response to determining that the position of the affine CU is not located at a specific position, wherein the specific position may include a corner position located at the lower right of the current CTU. For example, the lower right corner position of the current CTU (when a relative position value is stored) or the lower right corner position of the current picture / slice (when an absolute position value is stored) can be used to distinguish valid position information from invalid position information.

[0596] In some examples, an absolute position value or a relative position value may be used to indicate the position of the affine CU. In addition, the position may be stored in one of the following ways: horizontal coordinates and vertical coordinates with absolute position values; or horizontal coordinates and vertical coordinates with relative position values.

[0597] In some examples, the processor 2420 on the decoder side may obtain three pivot positions based on the position of the affine CU and the minimum size of the affine CU, obtain conventional motion information of three sub-blocks at the three pivot positions, and derive affine model information for the affine CU based on the conventional motion information of the three sub-blocks. In some examples, the three sub-blocks may include a first sub-block at the upper left of the affine CU having valid affine model information, a second sub-block that is a sub-block immediately to the right of the first sub-block, and a third sub-block that is a sub-block immediately below the first sub-block. Fig. 31C As shown, the three sub-blocks are indicated by 3103 respectively.

[0598] In some examples, processor 2420 at the decoder side may further obtain the CPMV for the affine CU by directly using the normal motion information stored for the three sub-blocks.

[0599] In some other examples, the processor 2420 at the decoder side may further derive a new affine model by using the regular motion information, and obtain the CPMV for the affine CU by using the new affine model.

[0600] Fig.34 is shown as Fig.33 The method for video decoding shown in FIG. 1 corresponds to a flowchart of a method for video encoding.

[0601] In step 3401, the processor 2420 on the encoder side may obtain a normalized position for an affine CU at a non-adjacent position based on a storage granularity, wherein the non-adjacent position is obtained by scanning a region that is not adjacent to the current CU.

[0602] like Fig.31AAs shown, each scanned non-adjacent position may be normalized / projected / cropped to the upper left position of the predefined / configured sub-block where the scanned non-adjacent position is located, for example, normalized / projected / cropped to a point close to 3101 or 3102 .

[0603] In step 3402, the processor 2420 on the encoder side may determine whether valid affine model information is stored for the normalized position, wherein the valid affine model information includes the position of the affine CU. Fig.31B As shown, only one piece of information is stored, namely the affine CU position, thus requiring lower memory storage.

[0604] In step 3403 , the processor 2420 at the encoder side may, in response to determining that valid affine model information is stored for the normalized position, derive affine model information for the affine CU based on the valid affine model information stored for the normalized position.

[0605] In some examples, the processor 2420 on the encoder side may determine that valid affine model information is stored for the normalized position in response to determining that the position of the affine CU is not located at a specific position, wherein the specific position may include a corner position located at the lower right of the current CTU. For example, the lower right corner position of the current CTU (when a relative position value is stored) or the lower right corner position of the current picture / slice (when an absolute position value is stored) can be used to distinguish valid position information from invalid position information.

[0606] In some examples, an absolute position value or a relative position value may be used to indicate the position of the affine CU. In addition, the position may be stored in one of the following ways: horizontal coordinates and vertical coordinates with absolute position values; or horizontal coordinates and vertical coordinates with relative position values.

[0607] In some examples, the processor 2420 on the encoder side may obtain three pivot positions based on the position of the affine CU and the minimum size of the affine CU, obtain conventional motion information of three sub-blocks at the three pivot positions, and derive affine model information for the affine CU based on the conventional motion information of the three sub-blocks. In some examples, the three sub-blocks may include a first sub-block at the upper left of the affine CU having valid affine model information, a second sub-block that is a sub-block immediately to the right of the first sub-block, and a third sub-block that is a sub-block immediately below the first sub-block. Fig. 31C As shown, the three sub-blocks are indicated by 3103 respectively.

[0608] In some examples, the processor 2420 at the encoder side may further obtain the CPMV for the affine CU by directly using the normal motion information stored for the three sub-blocks.

[0609] In some other examples, the processor 2420 at the encoder side may further derive a new affine model by using the conventional motion information, and obtain the CPMV for the affine CU by using the new affine model.

[0610] Fig.35 is a flowchart illustrating a method for video decoding according to an example of the present disclosure.

[0611] In step 3501 , the processor 2420 at the decoder side may scan a neighboring area of ​​the current CU to obtain a previously encoded neighboring affine CU, wherein the neighboring affine CU may include a plurality of internal sub-blocks.

[0612] In some examples, the neighboring affine CU may be located in an area adjacent or non-adjacent to the current CU.

[0613] In some examples, processor 2420 may be configured as discussed above. Fig.17A For example, the processor 2420 may scan the neighboring area of ​​the current CU based on a scanning rule including at least one scanning area, at least one scanning distance, and at least one scanning direction. The at least one scanning area may include a first scanning area and a second scanning area, the first scanning area may be determined according to a first maximum scanning distance, and the second scanning area may be determined according to a second maximum scanning distance, wherein the first maximum scanning distance indicates the maximum number of blocks from the first side of the current CU, the second maximum scanning distance indicates the maximum number of blocks from the second side of the current CU, and the first maximum scanning distance may be the same as or different from the second maximum scanning distance.

[0614] In addition, at least one scanning direction may include a first scanning direction and a second scanning direction, and the processor 2420 may predetermine a first maximum scanning distance or a second maximum scanning distance as a fixed value, and scan the first scanning area in the first scanning direction and scan the second scanning area in the second scanning direction.

[0615] In some examples, to constrain memory usage, when the selected non-adjacent affine CU is located within the current CTU or outside the current CTU, Fig.28Or the method proposed in 29 can be used to distinguish the acquisition of motion information of non-adjacent adjacent affine CUs. For example, the processor 2420 on the decoder side can obtain a restricted area that is not adjacent to the current CU based on the value associated with the restricted area, obtain the affine model information of the adjacent affine CU based on the restricted area, and obtain the motion information of multiple internal sub-blocks in the adjacent affine CU based on the affine model information of the adjacent affine CU. In some examples, the restricted area can be a predefined area associated with the current CU. This association can be a spatial relationship between the restricted area and the CU, or a predefined mapping relationship between the restricted area and the CU.

[0616] In some examples, when the selected non-adjacent affine CU is within the current CTU, the associated motion information may be obtained at different granularities and availability, and Figure 30A-Figure 30C and Figure 31A-Figure 31C The method provided in may be applied as an alternative. For example, the processor 2420 on the decoder side may obtain a normalized position for a neighboring affine CU based on the storage granularity, determine whether valid affine model information is stored for the normalized position, and in response to determining that valid affine model information is stored for the normalized position, derive the affine model information for the neighboring affine CU based on the valid affine model information stored for the normalized position.

[0617] In step 3502, the processor 2420 at the decoder side may derive a plurality of first affine model parameters based on neighboring affine CUs.

[0618] In some examples, the processor 2420 on the decoder side may derive multiple first affine model parameters based on the neighboring affine CU and the first linear regression model, wherein the input of the first linear regression model may include motion information of a first set of internal sub-blocks selected from multiple internal sub-blocks, and the number of the first set of internal sub-blocks may be less than or equal to the number of all multiple internal sub-blocks included in the neighboring affine CU.

[0619] In some examples, the processor 2420 on the decoder side may obtain three pivot positions based on at least one of the following information: the position of the adjacent affine CU, the width of the adjacent affine CU, or the height of the adjacent affine CU. In addition, the processor 2420 on the decoder side may obtain motion information of the three internal sub-blocks at the three pivot positions. In addition, the processor 2420 on the decoder side may derive multiple first affine model parameters based on the motion information of the three internal sub-blocks.

[0620] In step 3503, the processor 2420 on the decoder side may derive multiple second affine model parameters based on the second regression model, wherein the input of the second linear regression model may include motion information of a second internal sub-block set and motion information of multiple adjacent sub-blocks of the current CU, the second internal sub-block set may be selected from multiple internal sub-blocks, and the number of the second internal sub-block sets may be less than or equal to the number of all multiple internal sub-blocks included in the adjacent affine CU, and the multiple adjacent sub-blocks may be located in an adjacent area adjacent to the current CU.

[0621] In some examples, the adjacent region may include a top region and a left region, the top region may be above the current CU, and the left region may be to the left of the current CU and below the top region, the width of the top region may be equal to 2 times the width of the current CU, and the height of the left region may be equal to 1.5 times the height of the current CU, such as Fig.32 as shown in .

[0622] In step 3504, the processor 2420 at the decoder side may obtain a plurality of CPMVs based on a plurality of first affine model parameters and a plurality of second affine model parameters.

[0623] In some examples, the processor 2420 on the decoder side may obtain a ratio of the number of second internal sub-block sets to the number of multiple adjacent sub-blocks, and skip or terminate the derivation of multiple second affine model parameters in response to determining that the ratio is greater than a first threshold ratio or less than a second threshold ratio.

[0624] Fig.36 is shown as Fig.35 The method for video decoding shown in FIG. 1 corresponds to a flowchart of a method for video encoding.

[0625] In step 3601, the processor 2420 on the encoder side may scan a neighboring area of ​​the current CU to obtain a previously encoded neighboring affine CU, wherein the neighboring affine CU may include a plurality of internal sub-blocks.

[0626] In some examples, the neighboring affine CU may be located in an area adjacent or non-adjacent to the current CU.

[0627] In some examples, processor 2420 may be configured as discussed above. Fig.17AFor example, the processor 2420 may scan the neighboring area of ​​the current CU based on a scanning rule including at least one scanning area, at least one scanning distance, and at least one scanning direction. The at least one scanning area may include a first scanning area and a second scanning area, the first scanning area may be determined according to a first maximum scanning distance, and the second scanning area may be determined according to a second maximum scanning distance, wherein the first maximum scanning distance indicates the maximum number of blocks from the first side of the current CU, the second maximum scanning distance indicates the maximum number of blocks from the second side of the current CU, and the first maximum scanning distance may be the same as or different from the second maximum scanning distance.

[0628] In addition, at least one scanning direction may include a first scanning direction and a second scanning direction, and the processor 2420 may predetermine a first maximum scanning distance or a second maximum scanning distance as a fixed value, and scan the first scanning area in the first scanning direction and scan the second scanning area in the second scanning direction.

[0629] In some examples, to constrain memory usage, when the selected non-adjacent affine CU is located within the current CTU or outside the current CTU, Fig.28 Or the method proposed in 29 can be used to distinguish the acquisition of motion information of non-adjacent adjacent affine CUs. For example, the processor 2420 on the encoder side can obtain a restricted area that is not adjacent to the current CU based on the value associated with the restricted area, obtain the affine model information of the adjacent affine CU based on the restricted area, and obtain the motion information of multiple internal sub-blocks in the adjacent affine CU based on the affine model information of the adjacent affine CU. In some examples, the restricted area can be a predefined area associated with the current CU. This association can be a spatial relationship between the restricted area and the CU, or a predefined mapping relationship between the restricted area and the CU.

[0630] In some examples, when the selected non-adjacent affine CU is within the current CTU, the associated motion information may be obtained at different granularities and availability, and Figure 30A-Figure 30C and Figure 31A-Figure 31C The method provided in may be applied as an alternative. For example, the processor 2420 on the encoder side may obtain a normalized position for a neighboring affine CU based on the storage granularity, determine whether valid affine model information is stored for the normalized position, and in response to determining that valid affine model information is stored for the normalized position, derive the affine model information for the neighboring affine CU based on the valid affine model information stored for the normalized position.

[0631] In step 3602, the processor 2420 on the encoder side may derive a plurality of first affine model parameters based on a neighboring affine CU.

[0632] In some examples, the processor 2420 on the encoder side may derive multiple first affine model parameters based on the neighboring affine CU and the first linear regression model, wherein the input of the first linear regression model may include motion information of a first set of internal sub-blocks selected from multiple internal sub-blocks, and the number of the first set of internal sub-blocks may be less than or equal to the number of all multiple internal sub-blocks included in the neighboring affine CU.

[0633] In some examples, the processor 2420 on the encoder side may obtain three pivot positions based on at least one of the following information: the position of the adjacent affine CU, the width of the adjacent affine CU, or the height of the adjacent affine CU. In addition, the processor 2420 on the encoder side may obtain motion information of the three internal sub-blocks at the three pivot positions. In addition, the processor 2420 on the encoder side may derive multiple first affine model parameters based on the motion information of the three internal sub-blocks.

[0634] In step 3603, the processor 2420 on the encoder side may derive multiple second affine model parameters based on the second regression model, wherein the input of the second linear regression model may include motion information of a second internal sub-block set and motion information of multiple neighboring sub-blocks of the current CU, the second internal sub-block set may be selected from multiple internal sub-blocks, and the number of the second internal sub-block sets may be less than or equal to the number of all multiple internal sub-blocks included in the adjacent affine CU, and the multiple neighboring sub-blocks may be located in an adjacent area adjacent to the current CU.

[0635] In some examples, the adjacent region may include a top region and a left region, the top region may be above the current CU, and the left region may be to the left of the current CU and below the top region, the width of the top region may be equal to 2 times the width of the current CU, and the height of the left region may be equal to 1.5 times the height of the current CU, such as Fig.32 as shown in .

[0636] In step 3604, the processor 2420 at the encoder side may obtain a plurality of CPMVs based on a plurality of first affine model parameters and a plurality of second affine model parameters.

[0637] In some examples, the processor 2420 on the encoder side may obtain a ratio of the number of second internal sub-block sets to the number of multiple adjacent sub-blocks, and skip or terminate the derivation of multiple second affine model parameters in response to determining that the ratio is greater than a first threshold ratio or less than a second threshold ratio.

[0638] In some examples, a device for video decoding is provided. The device includes a processor 2420 and a memory 2440 configured to store instructions executable by the processor; wherein the processor is configured to perform the following when executing the instructions: Fig.33 and Fig.35Any of the methods shown in .

[0639] In some examples, a device for video encoding is provided. The device includes a processor 2420 and a memory 2440 configured to store instructions executable by the processor; wherein the processor is configured to perform the following when executing the instructions: Fig.34 and Fig.36 Any method shown.

[0640] In some other examples, a non-transitory computer-readable storage medium is provided having instructions stored therein. When the instructions are executed by the processor 2420, the instructions cause the processor to perform the following steps: Figure 33-Figure 36 In one example, multiple programs may be executed by a processor 2420 in a computing environment 2410 to receive (e.g., from Figure 1G The video encoder 20 in the computing environment 2410 receives a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames and / or one or more associated syntax elements, etc.), and can also be executed by the processor 2420 in the computing environment 2410 to perform the above-mentioned decoding method according to the received bitstream or data stream. In another example, multiple programs can be executed by the processor 2420 in the computing environment 2410 to perform the above-mentioned encoding method, encode the video information (e.g., video blocks representing video frames and / or one or more associated syntax elements, etc.) into a bitstream or data stream, and can also be executed by the processor 2420 in the computing environment 2410 to send a bitstream or data stream (e.g., to a Figure 2B Optionally, a non-transitory computer-readable storage medium may store therein a bit stream or a data stream, the bit stream or the data stream comprising a video decoder 30 in the encoder (e.g., Figure 1G The video encoder 20 in FIG. 1 generates a video signal for a decoder (eg, Figure 2B The non-transitory computer-readable storage medium may be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0641] By considering the specification and practice of the disclosure disclosed herein, other examples of the disclosure will be apparent to those skilled in the art. This application is intended to cover any variation, use or adaptation of the disclosure, which follows the general principles of the disclosure and includes such deviations from the disclosure within known or customary practice in the art. The specification and examples are intended to be considered exemplary only.

[0642] It will be understood that the present disclosure is not limited to the precise examples described above and shown in the accompanying drawings and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for video decoding, comprising: Obtaining, by a decoder based on a storage granularity, a normalized position for an affine coding unit CU at a non-adjacent position, wherein the non-adjacent position is obtained by scanning an area that is not adjacent to a current CU; Determining, by the decoder, whether valid affine model information is stored for the normalized position, wherein the valid affine model information includes the position of the affine CU; as well as In response to determining that the valid affine model information is stored for the normalized position, affine model information for the affine CU is derived by the decoder based on the valid affine model information stored for the normalized position.

2. The method according to claim 1, further comprising: In response to determining that the position of the affine CU is not located at a specific position, the decoder determines that the valid affine model information is stored for the normalized position, wherein the specific position includes a corner position located at the lower right of the current coding tree unit CTU.

3. The method according to claim 2, wherein the position of the affine CU is indicated using an absolute position value or a relative position value, and in, The position of the affine CU is stored in one of the following ways: horizontal and vertical coordinates with absolute position values; or Horizontal and vertical coordinates with relative position values.

4. The method according to claim 1, further comprising: Obtaining, by the decoder, three pivot positions based on the position of the affine CU and the minimum size of the affine CU; obtaining, by the decoder, conventional motion information of the three sub-blocks at the three pivot positions; as well as The affine model information for the affine CU is derived by the decoder based on the normal motion information of the three sub-blocks.

5. The method according to claim 4, wherein the three sub-blocks include: The first sub-block is located at the upper left of the affine CU having the valid affine model information, and the three sub-blocks further include a second sub-block which is a sub-block immediately to the right of the first sub-block and a third sub-block which is a sub-block immediately below the first sub-block.

6. The method according to claim 4, further comprising: The control point motion vector CPMV for the affine CU is obtained by the decoder by directly using the normal motion information stored for the three sub-blocks.

7. The method according to claim 4, further comprising: A new affine model is derived by the decoder by using the normal motion information, and a control point motion vector CPMV for the affine CU is obtained by the decoder by using the new affine model.

8. A method for video encoding, comprising: Obtaining, by an encoder based on a storage granularity, a normalized position for an affine coding unit CU at a non-adjacent position, wherein the non-adjacent position is obtained by scanning an area that is not adjacent to a current CU; Determining, by the encoder, whether valid affine model information is stored for the normalized position, wherein the valid affine model information includes the position of the affine CU; as well as In response to determining that the valid affine model information is stored for the normalized position, affine model information for the affine CU is derived by the encoder based on the valid affine model information stored for the normalized position.

9. The method according to claim 8, further comprising: In response to determining that the position of the affine CU is not located at a specific position, the encoder determines that the valid affine model information is stored for the normalized position, wherein the specific position includes a corner position located at the lower right of the current coding tree unit CTU.

10. The method according to claim 8, wherein the position of the affine CU is indicated using an absolute position value or a relative position value, and in, The position of the affine CU is stored in one of the following ways: horizontal and vertical coordinates with absolute position values; or Horizontal and vertical coordinates with relative position values.

11. The method according to claim 8, further comprising: Obtaining, by the encoder, three pivot positions based on the position of the affine CU and the minimum size of the affine CU; obtaining, by the encoder, conventional motion information of the three sub-blocks at the three pivot positions; as well as The affine model information for the affine CU is derived by the encoder based on the normal motion information of the three sub-blocks.

12. The method according to claim 11, wherein the three sub-blocks include: The first sub-block is located at the upper left of the affine CU having the valid affine model information, and the three sub-blocks further include a second sub-block which is a sub-block immediately to the right of the first sub-block and a third sub-block which is a sub-block immediately below the first sub-block.

13. The method according to claim 11, further comprising: The control point motion vector CPMV for the affine CU is obtained by the encoder by directly using the normal motion information stored for the three sub-blocks.

14. The method according to claim 11, further comprising: A new affine model is derived by the encoder by using the normal motion information, and a control point motion vector CPMV for the affine CU is obtained by the encoder by using the new affine model.

15. A method for video decoding, comprising: The decoder scans a neighboring area of ​​the current coding unit CU to obtain a previously encoded neighboring affine CU, wherein the neighboring affine CU includes a plurality of internal sub-blocks; The decoder derives a plurality of first affine model parameters based on the neighboring affine CU; The decoder derives a plurality of second affine model parameters based on a second linear regression model, wherein an input of the second linear regression model includes motion information of a second internal sub-block set and motion information of a plurality of neighboring sub-blocks of the current CU, wherein the second internal sub-block set is selected from the plurality of internal sub-blocks, and the number of the second internal sub-block set is less than or equal to the number of all the plurality of internal sub-blocks included in the neighboring affine CU, and wherein the plurality of neighboring sub-blocks are located in a neighboring region adjacent to the current CU; as well as A plurality of control point motion vectors CPMV are obtained by the decoder based on the plurality of first affine model parameters and the plurality of second affine model parameters.

16. The method according to claim 15, further comprising: The plurality of first affine model parameters are derived by the decoder based on the neighboring affine CU and a first linear regression model, wherein an input of the first linear regression model includes motion information of a first set of internal sub-blocks selected from the plurality of internal sub-blocks, wherein the number of the first set of internal sub-blocks is less than or equal to the number of all the plurality of internal sub-blocks included in the neighboring affine CU.

17. The method according to claim 16, further comprising: Obtaining, by the decoder, three pivot positions based on at least one of the following information: a position of the neighboring affine CU, a width of the neighboring affine CU, or a height of the neighboring affine CU; obtaining, by the decoder, motion information of three internal sub-blocks at the three pivot positions; as well as The plurality of first affine model parameters are derived by the decoder based on motion information of the three internal sub-blocks. The method according to claim 15 , wherein the neighboring affine CU is located in an area adjacent to or not adjacent to the current CU.

19. The method according to claim 15, wherein the adjacent region includes a top region and a left region, wherein the top region is above the current CU, and the left region is on the left side of the current CU and below the top region, wherein a width of the top region is equal to twice a width of the current CU, and wherein a height of the left region is equal to 1.5 times a height of the current CU.

20. The method of claim 15, further comprising: Scanning the neighboring area of ​​the current CU by the decoder based on a scanning rule, wherein the scanning rule includes at least one scanning area, at least one scanning distance, and at least one scanning direction; Wherein the at least one scanning area includes a first scanning area and a second scanning area, the first scanning area is determined according to a first maximum scanning distance, and the second scanning area is determined according to a second maximum scanning distance, wherein the first maximum scanning distance indicates the maximum number of blocks from the first side of the current CU, the second maximum scanning distance indicates the maximum number of blocks from the second side of the current CU, and the first maximum scanning distance is the same as or different from the second maximum scanning distance.

21. The method according to claim 20, wherein: The at least one scanning direction comprises a first scanning direction and a second scanning direction, The method further comprises: The first maximum scanning distance or the second maximum scanning distance is predetermined by the decoder as a fixed value; Scanning the first scanning area by the decoder in the first scanning direction; and The second scanning area is scanned by the decoder in the second scanning direction.

22. The method of claim 15, further comprising: Obtaining, by the decoder, the restricted area according to a value associated with the restricted area, wherein the restricted area is not adjacent to the current CU; Obtaining, by the decoder, affine model information of the neighboring affine CU based on the restricted area; and The motion information of the plurality of internal sub-blocks in the neighboring affine CU is obtained by the decoder based on the affine model information of the neighboring affine CU.

23. The method of claim 15, further comprising: Obtaining, by the decoder based on storage granularity, a normalized position for the neighboring affine CU; determining, by the decoder, whether valid affine model information is stored for the normalized position; as well as In response to determining that the valid affine model information is stored for the normalized position, affine model information for the neighboring affine CU is derived by the decoder based on the valid affine model information stored for the normalized position.

24. The method of claim 15, further comprising: Obtaining, by the decoder, a ratio of the number of the second internal sub-block sets to the number of the plurality of adjacent sub-blocks; as well as In response to determining that the ratio is greater than a first threshold ratio or less than a second threshold ratio, derivation of the plurality of second affine model parameters is skipped or terminated by the decoder.

25. A method for video encoding, comprising: The encoder scans a neighboring area of ​​the current coding unit CU to obtain a previously encoded neighboring affine CU, wherein the neighboring affine CU includes a plurality of internal sub-blocks; The encoder derives a plurality of first affine model parameters based on the neighboring affine CU; The encoder derives a plurality of second affine model parameters based on a second linear regression model, wherein an input of the second linear regression model includes motion information of a second internal sub-block set and motion information of a plurality of neighboring sub-blocks of the current CU, wherein the second internal sub-block set is selected from the plurality of internal sub-blocks, and the number of the second internal sub-block set is less than or equal to the number of all the plurality of internal sub-blocks in the neighboring affine CU, and wherein the plurality of neighboring sub-blocks are located in a neighboring region adjacent to the current CU; as well as A plurality of control point motion vectors CPMV are obtained by the encoder based on the plurality of first affine model parameters and the plurality of second affine model parameters.

26. The method according to claim 25, further comprising: The plurality of first affine model parameters are derived by the encoder based on the neighboring affine CU and a first linear regression model, wherein an input of the first linear regression model includes motion information of a first set of internal sub-blocks selected from the plurality of internal sub-blocks, wherein the number of the first set of internal sub-blocks is less than or equal to the number of all the plurality of internal sub-blocks included in the neighboring affine CU.

27. The method according to claim 26, further comprising: Obtaining, by the encoder, three pivot positions based on at least one of the following information: a position of the neighboring affine CU, a width of the neighboring affine CU, or a height of the neighboring affine CU; obtaining, by an encoder, motion information of three internal sub-blocks at the three pivot positions; and The plurality of first affine model parameters are derived by the encoder based on motion information of the three internal sub-blocks. The method according to claim 25 , wherein the neighboring affine CU is located in an area adjacent to or not adjacent to the current CU.

29. The method according to claim 25, wherein the adjacent region includes a top region and a left region, wherein the top region is above the current CU, and the left region is on the left side of the current CU and below the top region, wherein a width of the top region is equal to twice a width of the current CU, and wherein a height of the left region is equal to 1.5 times a height of the current CU.

30. The method of claim 25, further comprising: Scanning, by the encoder, the neighboring area of ​​the current CU based on a scanning rule, wherein the scanning rule includes at least one scanning area, at least one scanning distance, and at least one scanning direction; The at least one scanning area includes a first scanning area and a second scanning area, the first scanning area is determined according to a first maximum scanning distance, and the second scanning area is determined according to a second maximum scanning distance, wherein the first maximum scanning distance indicates the maximum number of blocks from the first side of the current CU, the second maximum scanning distance indicates the maximum number of blocks from the second side of the current CU, and the first maximum scanning distance is the same as or different from the second maximum scanning distance.

31. The method according to claim 30, wherein: The at least one scanning direction comprises a first scanning direction and a second scanning direction, The method further comprises: The first maximum scanning distance or the second maximum scanning distance is predetermined by the encoder as a fixed value; Scanning the first scanning area in the first scanning direction by the encoder; and The second scanning area is scanned by the encoder in the second scanning direction.

32. The method of claim 25, further comprising: Obtaining, by the encoder, the restricted area according to a value associated with the restricted area, wherein the restricted area is not adjacent to the current CU; Obtaining, by the encoder, affine model information of the neighboring affine CU based on the restricted area; and The encoder obtains motion information of the plurality of internal sub-blocks in the neighboring affine CU based on the affine model information of the neighboring affine CU.

33. The method of claim 25, further comprising: Obtaining, by the encoder, a normalized position for the neighboring affine CU based on storage granularity; Determining, by the encoder, whether valid affine model information is stored for the normalized position; as well as In response to determining that the valid affine model information is stored for the normalized position, affine model information for the neighboring affine CU is derived by the encoder based on the valid affine model information stored for the normalized position.

34. The method of claim 25, further comprising: Obtaining, by the encoder, a ratio of the number of the second internal sub-block set to the number of the plurality of adjacent sub-blocks; as well as In response to determining that the ratio is greater than a first threshold ratio or less than a second threshold ratio, derivation of the plurality of second affine model parameters is skipped or terminated by the encoder.

35. An apparatus for video decoding, comprising: one or more processors; as well as a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors, Wherein the one or more processors are configured to perform the method according to any one of claims 1-7 and 15-24 when executing the instructions.

36. An apparatus for video encoding, comprising: one or more processors; as well as a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors, Wherein the one or more processors are configured to perform the method according to any one of claims 8-14 and 25-34 when executing the instructions.

37. A non-transitory computer-readable storage medium for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to receive a bit stream and perform the method of any one of claims 1-7 and 15-24 based on the bit stream.

38. A non-transitory computer-readable storage medium for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to perform the method according to any one of claims 8-14 and 25-34 to encode a current CU into a bitstream and send the bitstream.

39. A non-transitory computer-readable storage medium for storing a bit stream to be decoded by the method of any one of claims 1-7 and 15-24.

40. A non-transitory computer-readable storage medium storing a bitstream generated by the method of any one of claims 8-14 and 25-34.

Images