Method, device and related computer readable storage medium for video data coding

Abstract

Methods and apparatuses related to video data coding, and computer readable media are described herein. It is determined to use affine prediction to code a current block of video data. A plurality of candidates for a candidate list for the current block is generated, including: when it is determined that one or more inherited candidates are available, determining that each inherited candidate inherits an affine model of an associated neighboring block; and when it is determined that one or more structural candidates are available, deriving the one or more structural candidates, wherein each structural candidate is derived based on a plurality of associated neighboring blocks of the current block. The current candidate list is filled by generating other candidates for the candidate list when it is determined that the candidate list is not yet full.

Description

[0001] Related applications

[0002] This application is a divisional application of the invention patent with application number 201980053534.1 and invention title "Method, Apparatus and Related Computer-Readable Storage Medium for Video Data Encoding and Decoding". Technical Field

[0003] This invention generally relates to video encoding and decoding. Specifically, this invention relates to generating a candidate list of affine motion vectors for padding the candidate list. Background Technology

[0004] Various video coding technologies can be used for video encoding and decoding, such as for data storage and / or transmission. Video coding technologies can also provide the means to decode encoded video for playback. A video codec can contain electronic circuitry and / or software for compressing and / or decompressing digital video. Multiple video coding standards exist, and video codecs typically need to conform to one or more standards. For example, High Efficiency Video Codec (HEVC) is an international video coding standard developed by the Joint Video Coding Panel (JCT-VC). Another example is Universal Video Codec (VVC), another international video coding standard developed by the Joint Video Experts Group (JVET). Many video coding standards, including HEVC and VVC, use spatial and temporal compression techniques. Compression processes encode video data, producing a resulting bitstream by generating residual data that can be transformed, quantized, and entropy-coded. Similarly, the decompression program performs entropy decoding, inverse quantizing, and inverse transforming on the bitstream to reconstruct the residual data and ultimately produce the image. Summary of the Invention

[0005] Some embodiments of this invention disclose apparatus, systems, and methods for generating motion vector candidates for an affine motion vector candidate list, including generating candidates according to one or more techniques. Some embodiments employ techniques for filling the affine motion vector candidate list, including generating one or more candidates in the list after adding inherited affine candidates and structural affine candidates. Other embodiments employ techniques including generating one or more zero motion vector candidates, deriving affine candidates, and / or temporal candidates.

[0006] Some embodiments relate to video data encoding / decoding methods, including: determining a current block of video data to be encoded using affine prediction; generating a plurality of candidates for a candidate list of the current block, including: when it is determined that one or more inheritance candidates are available, determining that each inheritance candidate inherits an affine model of an associated neighboring block; and when it is determined that one or more structural candidates are available, deriving the one or more structural candidates, wherein each structural candidate is derived based on a plurality of associated neighboring blocks of the current block; and when it is determined that the candidate list is not yet full, filling the current candidate list by generating other candidates for the candidate list, including... When one or more additional derivation candidates are determined to be available, the one or more additional derivation candidates are generated based on multiple associated neighboring blocks of the current block, and the generated one or more additional derivation candidates are added to the candidate list; when it is determined that the candidate list is not yet full, one or more temporal motion vector candidates are generated based on the temporal merging image, and the generated one or more temporal motion vector candidates are added to the candidate list; and when the candidate list is not yet full after being filled with the one or more additional derivation candidates, the candidate list is filled by deriving zero motion vector candidates that need to be filled into the candidate list. In some examples, if the affine prediction is associated with a four-parameter affine pattern, each candidate in the candidate list includes a first motion vector about a first control point and a second motion vector about a second control point; and if the affine prediction is associated with a six-parameter affine pattern, each candidate in the candidate list includes the first and second motion vectors, and a third motion vector about a third control point.

[0007] In some examples, the second set of candidates for the candidate list is generated by: generating one or more additional derivation candidates based on multiple associated nearby blocks of the current block, and adding the generated one or more additional derivation candidates to the candidate list; when it is determined that the candidate list is not yet full, generating the temporal motion vector candidate based on the temporal merging image, and adding the generated temporal motion vector candidate to the candidate list; and when it is determined that the candidate list is not yet full, generating the remaining number of zero motion vector candidates that need to be filled into the candidate list.

[0008] In some instances, the method of generating one or more additional derivation candidates based on multiple associated neighboring blocks of the current block includes: determining the available motion vectors based on a first set of neighboring blocks; and generating a first candidate that includes the first available motion vector, and generating it for each control point of the first candidate.

[0009] In some examples, the method includes determining that the candidate list is not yet full; determining a second available motion vector based on a second set of neighborhood blocks, the second set of neighborhood blocks being different from the first set of neighborhood blocks; and generating a second candidate that includes the second available motion vector, and generating it for each control point of the second candidate.

[0010] In some examples, the first group of nearby blocks is near a first sub-block of the current block associated with a top-left control point, and the second group of nearby blocks is a second sub-block of the current block adjacent to a top-right control point; or the first group of nearby blocks is near the first sub-block, and the second group of nearby blocks is near a third sub-block of the current block associated with a bottom-left control point; or the first group of nearby blocks is near the third sub-block, and the second group of nearby blocks is near the second sub-block; or the first group of nearby blocks is near the third sub-block, and the second group of nearby blocks is near the first sub-block; or the first group of nearby blocks is near the second sub-block, and the second group of nearby blocks is near the first sub-block; or the first group of nearby blocks is near the second sub-block, and the second group of nearby blocks is near the third sub-block.

[0011] In some examples, the method includes determining that the candidate list is not yet full; determining a third available motion vector based on a third group of neighborhood blocks, which is different from the first and second groups of neighborhood blocks; and generating a third candidate that includes the third available motion vector, and generating it for each control point of the third candidate.

[0012] In some examples, the first group of nearby blocks is near the first child block of the current block associated with the top-left control point, the second group of nearby blocks is near the second child block of the current block associated with the top-right control point, and the third group of nearby blocks is near the third child block of the current block associated with the bottom-left control point; or the first group of nearby blocks is near the first child block, the second group of nearby blocks is near the third child block, and the third group of nearby blocks is near the second child block; or the first group of nearby blocks is near the third child block, and the second group of nearby blocks is near the second child block. The third group of nearby blocks is located near the first sub-block; or the first group of nearby blocks is located near the third sub-block, the second group of nearby blocks is located near the first sub-block, and the third group of nearby blocks is located near the second sub-block; or the first group of nearby blocks is located near the second sub-block, the second group of nearby blocks is located near the first sub-block, and the third group of nearby blocks is located near the third sub-block; or the first group of nearby blocks is located near the second sub-block, the second group of nearby blocks is located near the third sub-block; and the third group of nearby blocks is located near the first sub-block.

[0013] In some examples, generating the temporal motion vector candidate includes: determining from a block of the temporally merging image that a motion vector is available, the block being the lower right sub-block of the temporally merging block adjacent to the current block in the temporally merging image; and generating the temporal motion vector candidate that includes the motion vector, and generating it for each control point of the temporal motion vector candidate.

[0014] In some examples, the method includes determining from a block of the time-panned image that a motion vector is unavailable, the block being a lower right sub-block of a time-panned block adjacent to the current block in the time-panned image; and generating the time motion vector candidate including the motion vector from the lower right sub-block of the time-panned block, and generating it for each control point of the time motion vector candidate.

[0015] In some cases, generating the temporal motion vector candidate includes: determining the motion vector of the temporal motion vector candidate based on the temporal juxtaposition image; and scaling the motion vector before generating the temporal motion vector candidate.

[0016] In some examples, generating the one or more zero motion vector candidates includes: determining a number of reference images associated with the candidate list; and generating the number of zero motion vector candidates, wherein each generated zero motion vector candidate includes a separate reference index, which is selected from a set of integers from zero to the number minus one.

[0017] In some examples, the method includes determining that the candidate list is not yet full; generating a remaining number of zero motion vector candidates to be added to the candidate list, wherein the reference index of each of the remaining number of zero motion vector candidates is zero.

[0018] In some cases, the second set of candidates that generates the candidate list includes those that use the same motion vector for each control point.

[0019] Some implementations relate to apparatus for encoding and decoding video data. The apparatus includes a processor that communicates with memory, executing instructions stored in the memory such that the processor: determines a current block of video data to be encoded using affine prediction; generates a first set of candidates for a candidate list of the current block, including: determining one or more inherited candidates, each inheriting an affine model of an associated neighboring block; and deriving one or more structural candidates, each structural candidate derived based on multiple associated neighboring blocks of the current block. The instructions in memory can also be configured to cause the processor to: after the first set of candidates is generated, determine whether the candidate list is full; if the candidate list is not full, fill the candidate list by generating a second set of candidates, including one or more of the following: generating one or more zero motion vector candidates; generating one or more additional derived candidates based on multiple associated neighboring blocks of the current block; and generating temporal motion vector candidates based on a temporally juxtaposed image.

[0020] In some examples, the second set of candidates for generating the candidate list includes: generating one or more additional derived candidates based on multiple associated nearby blocks of the current block, and adding the generated one or more additional derived candidates to the candidate list; when it is determined that the candidate list is not yet full, generating the temporal motion vector candidate based on the temporal merging image, and adding the generated temporal motion vector candidate to the candidate list; and when it is determined that the candidate list is not yet full, generating a remaining number of zero motion vector candidates that need to be filled into the candidate list.

[0021] In some instances, the one or more additional derivation candidates are generated based on a plurality of associated neighboring blocks of the current block, including: determining the available motion vectors based on a first set of neighboring blocks; and generating a first candidate that includes the first available motion vector, and generating it for each control point of the first candidate.

[0022] In some examples, the method of generating the temporal motion vector candidate includes: determining from a block of the temporally merging image that a motion vector is available, the block being the lower right sub-block of the temporally merging block adjacent to the current block in the temporally merging image; and generating the temporal motion vector candidate that includes the motion vector, and generating it for each control point of the temporal motion vector candidate.

[0023] In some cases, a motion vector is determined to be unavailable from a block of the time-panned image, which is the lower right sub-block of the time-panned image adjacent to the current block in the time-panned image; and from the lower right sub-block of the time-panned image, a time motion vector candidate including the motion vector is generated, and this is generated for each control point of the time motion vector candidate.

[0024] Some embodiments relate to at least one non-transitory computer-readable storage medium for storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the following actions: determine a current block of video data to be encoded using affine prediction; generate a plurality of candidates for a candidate list of the current block, including: when it is determined that one or more inheritance candidates are available, determining that each inheritance candidate inherits an affine model of an associated neighboring block; and when it is determined that one or more structural candidates are available, deriving the one or more structural candidates, wherein each structural candidate is derived based on a plurality of associated neighboring blocks of the current block; and when it is determined that the candidate list has not yet been generated... When the current candidate list is full, it is populated by generating other candidates, including: when it is determined that one or more additional derived candidates are available, generating one or more additional derived candidates based on multiple associated nearby blocks of the current block, and adding the generated one or more additional derived candidates to the candidate list; when it is determined that the candidate list is not yet full, generating one or more time motion vector candidates based on the time-partitioned image, and adding the generated one or more time motion vector candidates to the candidate list; and when the candidate list is not yet full after being populated by the one or more additional derived candidates, the candidate list is populated by deriving zero motion vector candidates that need to be populated. Attached Figure Description

[0025] In the numerous accompanying drawings, each identical or nearly identical component is represented by a similar symbol. For clarity, not every component is labeled in every drawing. The drawings are not necessarily drawn to scale, but rather the focus is on clearly illustrating the various aspects of the technology and apparatus described herein.

[0026] Figure 1 An exemplary video encoding configuration is shown according to some embodiments.

[0027] Figure 2 Some embodiments show details of an exemplary video encoder.

[0028] Figure 3 Details of an exemplary video decoder are shown in some embodiments.

[0029] Figure 4 Some embodiments show examples of two control point motion vectors V0 and V1.

[0030] Figure 5The icon, according to some embodiments, shows examples of neighbor reconstructed blocks, which can be used to generate a candidate MVP pairing list for affine inter-mode.

[0031] Figure 6 The icon, according to some embodiments, shows an affine block using a four-parameter affine pattern, individually showing exemplary positions of control points v0 and v1.

[0032] Figure 7 The icon, according to some embodiments, shows an example where the MVs of two control points v0 and v1 are stored in the center of the upper left and upper right sub-blocks.

[0033] Figure 8 The icon, as shown in some embodiments, illustrates an example of changing the position of the control point in four-parameter affine mode.

[0034] Figure 9 The icon, as shown in some embodiments, indicates the position of the control point of the affine block using the six-parameter affine mode.

[0035] Figure 10 The icon, according to some embodiments, shows an example of the position of the control point after modification using the six-parameter affine mode.

[0036] Figure 11 The icon, according to some embodiments, shows an example of redundant coefficient coding.

[0037] Figure 12 The icon, according to some embodiments, shows an example where only a portion of the coefficients in the coefficient block is scanned.

[0038] Figure 13 The icons, based on some examples, show affine code blocks B and E.

[0039] Figure 14 The icon, according to some examples, shows that additional MVs can come from time and bit MVs, including Col-BR, Col-H, Col-BL, Col-A1, Col-A0, Col-B0, Col-B1, or Col-TR.

[0040] Figure 15 The icon, in some examples, shows the positions of the control points at the four corners of the current block.

[0041] Figure 16 The icon, in some embodiments, indicates the control point location of the sub-block.

[0042] Figure 17 The icon, in some embodiments, shows a control point located at the center of the four corner sub-blocks.

[0043] Figure 18A The -B icon indicates the location of the MV in some implementations.

[0044] Figure 19 According to some embodiments of the technology described herein, a computerized method for generating a candidate list is illustrated in flowchart form. Detailed Implementation

[0045] Many techniques exist for generating candidate lists for various prediction techniques, such as affine prediction, including affine merge mode and / or affine inter mode. For example, candidates in affine prediction may include inherited affine candidates that inherit the affine mode of neighboring blocks (e.g., control point motion vectors will therefore come from the same neighboring blocks) and / or structural candidates whose control point motion vectors will come from different neighboring blocks. The inventors have discovered and recognized that existing candidate list generation techniques may not be able to completely fill the candidate list; when the candidate list is not full, it can cause problems for encoding and / or decoding. For example, when the candidate list is not full, the encoder or decoder may select a candidate that does not exist.

[0046] Therefore, the inventors have developed techniques to generate candidates for a candidate list. For example, the developed techniques can be used when the candidate list is not full after any affine candidate and / or structural candidate has been added. In some embodiments, the techniques may include generating one or more zero-motion-vector candidates to fill the candidate list. In some embodiments, the techniques may include generating one or more additional derived candidates, wherein each candidate includes the same motion vector for each control point. In some embodiments, the techniques may include generating one or more temporal candidates that use temporal motion vectors (MVs) from a temporal picture. In one embodiment, if the candidate list is not full after any affine candidate and / or structural candidate has been added, additional derived candidates generated based on multiple associated neighboring blocks of the current block may first be used to fill the candidate list. If the candidate list is still not full after additional derived candidates have been added, temporal candidates generated based on temporal parallel blocks of the current block may be used to fill the candidate list. However, if the candidate list is still not full after temporal candidates have been added, the remaining portion of the candidate list is filled with zero-motion-vector candidates. In another embodiment, if the candidate list is not full after any affine candidate and / or structural candidate is added, then time candidates generated based on the time-interval block of the current block can be used to fill the candidate list first. If the candidate list is still not full after time candidates are added, then additional derivation candidates generated based on multiple associated neighboring blocks of the current block can be used to fill the candidate list. However, if the candidate list is still not full after additional derivation candidates are added, then the remaining portion of the candidate list is filled with zero motion vector candidates.

[0047] In some embodiments, the techniques described herein may be used to fill / fill the candidate list. For example, in some embodiments, when any affine candidate and / or structural candidate is added, the techniques may include: (a) adding a candidate with a first available motion vector to a set of neighboring blocks adjacent to a sub-block of an associated lower-left control point; (b) adding a candidate with a first available motion vector to a set of neighboring blocks adjacent to a sub-block of an associated upper-right control point; (c) adding a candidate with a first available motion vector to a set of neighboring blocks adjacent to a sub-block of an associated upper-left control point; (d) adding a candidate with a temporal motion vector (MV), the MV being derived from a juxtaposed block or image; and (e) adding one or more zero motion vector candidates.

[0048] In the following description, more specific details are given regarding the systems and methods disclosed, as well as the operating environment in which they operate, in order to provide a thorough understanding of the disclosed subject matter. Furthermore, the examples provided below should be understood as exemplary, and other systems and methods may be conceived within the scope of this disclosure.

[0049] Figure 1 According to some embodiments, an exemplary video encoding apparatus 100 is shown. The video encoding apparatus 100 includes one or more cameras 102A-102N, collectively referred to as camera 102, for generating video data 106, which is provided to an encoder 108. Camera 102 can be any type of camera, including those with recording capabilities and / or separate camera and recording functions. A decoding apparatus 110 is used to receive the encoded data. The decoding apparatus 110 can receive video / video products (e.g., digital video discs or other computer-readable media) via a broadcast network, a mobile network (e.g., a cellular network), and / or the Internet. The decoding apparatus 110 can be part of a computer, a head-mounted display device, or any other device with decoding capabilities. The decoding apparatus 110 includes a decoder 112 configured to decode the encoded video to generate video data 106, and a display 116 is used to display the video data 114.

[0050] Figure 2 According to some embodiments, details of an exemplary video encoder 200 are shown. The video encoder 200 is used to receive input video 202 and includes a prediction unit 204, a conversion and quantization unit 206, a CABAC unit 208, an inverse conversion and quantization unit 210, a deblocking unit 212, a SAO unit 214, a decoded picture buffer 216, and an SAO parameter estimation unit 218, or at least some of the above components.

[0051] The prediction unit 204 includes inter-frame prediction processing 220 (which further includes a motion estimation unit 222 and a motion compensation unit 224) and intra-frame prediction processing 226 (which further includes an intra-frame estimation unit 228 and an intra-frame prediction unit 230). In other embodiments, the video encoder 200 may include... Figure 2 More, fewer, and / or different components as described in the document.

[0052] For block-based motion compensation, the basic unit of compensation is a CU (Cubic Unit), which can be a 2Nx2N square block. Each CU can be recursively divided into four smaller CUs, until a predetermined minimum block size is reached. Each CU can include one or more PUs (Programming Components). If a CU contains only one PU, then the PU can be considered the same as the CU. The video encoder 200 can encode the CUs. The prediction unit 204 can also divide the CU into one or more PUs, which is part of the CU encoding process.

[0053] Prediction unit 204 can use inter-frame prediction unit 220 to perform inter-frame prediction. This inter-frame prediction process may include generating prediction data for each PU, which can be done for each PU of a CU. Depending on the type of inter-frame prediction, motion estimation unit 222 can search for reference samples from decoded image buffer 216, including reference samples in list 0 or list 1. Motion estimation unit 222 can generate reference image indices for positions in list 0 or list 1 and can generate motion vectors to indicate displacements between the reference location and a sample block of the PU. Motion compensation unit 224 can generate predicted sample blocks for the PU, which can be generated at least partially based on actual samples or interpolated samples in the reference region indicated by the PU's motion vectors.

[0054] Prediction unit 204 may use intra-prediction unit 226 to perform intra-prediction. The intra-prediction process may include generating prediction data for a prediction unit (PU), which can be used to perform intra-prediction on that PU. The prediction data for the PU may include the prediction block of the PU and various syntax elements. Prediction unit 204 may select the prediction data for the PU, that is, it may select it from the prediction data generated by the inter-prediction process or the prediction data generated by the intra-prediction process. In some embodiments, prediction unit 204 may include a multi-hypothesis (MH) mode (discussed later). In some embodiments, the MH mode may be an independent mode separate from inter- or intra-prediction. In some embodiments, the MH mode may be a part of inter- or intra-prediction. For example, the MH mode (considered an inter-prediction mode) may add additional prediction hypotheses on top of existing prediction hypotheses derived from the inter-prediction mode. As another example, the MH mode (considered an intra-prediction mode) may add additional prediction hypotheses on top of existing prediction hypotheses derived from the intra-prediction mode.

[0055] The conversion and quantization unit 206 can generate conversion coefficients for each TU using a conversion mode (e.g., Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), or other types of conversion), and quantize these conversion coefficients. The CABAC unit 208 performs entropy coding on the quantized conversion coefficients and / or any other side information to generate a bitstream. In other embodiments, the CABAC unit 208 can be replaced by other entropy coding units to generate entropy-coded data. The video encoder 200 can output the entropy-coded conversion coefficients in the bitstream.

[0056] The inverse transformation and quantization unit 210 can apply inverse quantization and inverse transformation modes (e.g., inverse discrete cosine transform, inverse discrete sine transform, or other types of inverse transform) to reconstruct the aresidual block from the coefficient block. The deblocking unit 212 can perform one or more deblocking operations on the reconstructed block. The SAO unit 214 and the SAO parameter estimation unit 218 perform sample adaptive offset (SAO), which is an in-loop filtering technique. The SAO procedure may include: classifying the reconstructed samples into different categories, obtaining an offset for each category, and then adding the offset to each sample of the aforementioned categories. The offset for each category can be signaled to the decoder to reduce sample distortion.

[0057] The decoded image buffer 216 can store the reconstructed and SAO-processed blocks. As described herein, the prediction unit 204 can use the reconstructed blocks (reconstructed blocks) for inter-frame prediction and / or intra-frame prediction.

[0058] Figure 3 According to some embodiments, details of an exemplary video decoder 300 are shown. The video decoder 300 includes an entropy decoding unit 302, a prediction unit 304, an inverse quantization and inverse conversion unit 306, a reconstruction unit 308, a deblocking filter 310, a SAO unit 312, a reference picture and buffer 314, and an SAO parameter unit 316, or at least some of the above components. For illustrative purposes, Figure 3 This discussion uses HEVC as an example; however, these techniques can also be applied to other video coding standards. In other embodiments, the video decoder 300 may also include... Figure 3 More, fewer, and / or different components as described in the document.

[0059] Entropy decoding unit 302 parses the bitstream to decode syntax elements. Prediction unit 304 can construct one or more reference image lists (e.g., List0 and / or List1), for example, using syntax elements indicated in the bitstream. Prediction unit 304 can perform motion compensation and / or intra-frame prediction. For example, if a PU is encoded using inter-frame prediction, prediction unit 304 can extract the motion information of the PU and then use it to determine one or more reference regions of the PU. Based on sample blocks of one or more of these reference regions, prediction unit 304 can generate prediction blocks for the PU. As another example, if a PU is encoded using intra-frame prediction, prediction unit 304 can generate prediction blocks for the PU by performing intra-frame prediction based on neighboring PUs. In some embodiments, prediction unit 304 may include MH (multi-hypothesis) mode processing (discussed later herein). In some embodiments, MH mode processing may be an independent mode separate from inter-frame or intra-frame prediction processing. In some embodiments, the MH mode can be part of either inter-frame or intra-frame prediction. For example, the MH mode (when considered as an inter-frame prediction mode) may add an additional prediction assumption on top of an existing prediction assumption derived from the inter-frame prediction mode. As another example, the MH mode (when considered as an intra-frame prediction mode) may add an additional prediction assumption on top of an existing prediction assumption derived from the intra-frame prediction mode.

[0060] The inverse quantization and inverse transformation unit 306 can inverse quantize a coefficient block and apply an inverse transformation to generate a residual block. The reconstruction unit 308 can reconstruct these coding blocks.

[0061] The reconstruction unit 308 can reconstruct these code blocks of the CU (e.g., luma, Cb, and Cr code blocks). The reconstruction can be performed using transform blocks associated with the CU's TU (e.g., luma, Cb, and Cr transform blocks) and prediction blocks associated with the CU's PU (e.g., luma, Cb, and Cr prediction blocks). Depending on the application, inter-frame prediction data and / or intra-frame prediction data can be used. For example, the reconstruction unit 308 can add samples (residual components) of the transform blocks (e.g., luma, Cb, and Cr transform blocks) to samples (predictor components) of the corresponding prediction blocks to reconstruct these code blocks of the CU.

[0062] Deblocking filter 310 performs deblocking to reduce blocking artifacts in the corresponding code block. SAO unit 312 and SAO parameter unit 316 implement SAO in-loop filtering. Offsets of each SAO type can be sent to decoder 300, which can use them to reduce sample distortion. Reference image and buffer 314 can store the generated deblocked and SAO-processed blocks. Buffer 314 can provide reference images for subsequent motion compensation, intra-frame prediction, and picture presentation, as shown in Figure 318.

[0063] To achieve optimal encoding and decoding efficiency in the hybrid code architecture of HEVC, each PU has two prediction modes: intra-frame prediction mode and inter-frame prediction mode. In intra-frame prediction mode, spatially neighboring reconstructed pixels can be used to generate directional predictions. In inter-frame prediction mode, temporally reconstructed reference frames can be used to generate motion-compensated predictions. There are three different modes: Skip, Merge, and Inter-Advanced Motion Vector Prediction (Inter AMVP).

[0064] When the PU is encoded in Inter AMVP mode, motion compensation prediction is performed using transmitted motion vector differences (MVDs), which are used in conjunction with motion vector predictors (MVPs) to derive motion vectors (MVs). To determine the MVP in Inter AMVP mode, an advanced motion vector prediction architecture is used to select the motion vector predictors from an AMVP candidate group comprising two spatial MVPs and one temporal MVP. Therefore, in AMVP mode, the MVP index and corresponding MVDs of the MVPs are encoded and transmitted. Furthermore, the inter-frame prediction directions (indicating the prediction direction between bidirectional and unidirectional prediction, i.e., lists List0 (L0) and List1 (L1)) and the reference frame index accompanying each list are also encoded and transmitted.

[0065] When the CU / PU is encoded in skip or merge mode, no motion information is transmitted except for the selected candidate merge index. This is because skip and merge modes use motion inference (MV = MVP + MVD, where MVD is zero) to obtain motion information from spatially nearby blocks (spatial candidates) or temporally nearby blocks (temporal candidates) in the co-located picture, where the co-located picture is the first reference picture in list 0 or list 1 sent by the slice header. If it is a skip PU, residual signals are also omitted. To determine the merge index for skip and merge modes, a merge architecture is used to select motion vector predictors from a merge candidate group consisting of four spatial MVPs and one temporal MVP.

[0066] Four-parameter affine prediction has been proposed and can be used for affine merging and / or affine inter-frame modes. Detailed explanations can be found in the relevant ITU-T 13-SG16-C-1016 standard document "Affine transform prediction for next generation video coding," proposed by Sixin Lin, Huanbang Chen, Hong Zhang, Sychev Maxim, Haitao Yang, and Jiantong Zhou, ITU-T 13-SG16-C-1016, ITU-T Standardization Sector, Study Group 16, Question Q6 / 16, Contribution 1016, Geneva, Switzerland, September 2015. The above-listed content is incorporated herein by reference.

[0067] When the affine block is moving, the block's motion vector field (MV field) can be described by the motion vectors of two control points or four parameters according to the following Equation 1:

[0068]

[0069] Figure 4 Some embodiments show an example 400 with two control point motion vectors V0 and V1. As shown, the transition block 402 is a rectangular block, and the MV field at each point of this moving block can be described by the following equation 2:

[0070]

[0071] Where (x, y) represents a point of the moving block, (v 0x ,v 0y ) is the motion vector V0 of the control point at the top left corner of the block, and (v 1x ,v 1y ) is the motion vector V1 of the control point in the upper right corner of the block.

[0072] In some embodiments, affine motion prediction can be used in inter-frame mode. Figure 5 According to some embodiments, an example of adjacent reconstructed blocks is shown as icon 500, which can be used to generate a candidate MVP pairing list for affine inter-frame modes. In some embodiments, for CUs encoded in inter-frame modes, when the CU size is equal to or greater than 16x16, an affine flag can be sent to indicate whether an affine inter-frame mode has been applied. In some embodiments, if the current CU is encoded in affine inter-frame modes, a candidate MVP pairing list can be established using adjacent valid reconstructed blocks. Figure 5As shown, taking the current CU502 as an example, v0 is a motion vector selected from blocks A0, A1, or A2, while v1 is a motion vector selected from blocks B0 and B1. The index of the candidate MVP pairing can be sent in the bitstream, and the MV difference (MVD) of the two control points can be encoded in the bitstream.

[0073] In some embodiments, affine motion prediction can be used in merging mode. If the current CU is a merging PU, then the adjacent 5 blocks ( Figure 5 The system uses blocks C0, B0, B1, C1, and A0 to determine whether one of these neighboring blocks is in affine inter-frame mode or affine merging mode. If so, an affine flag can be sent to indicate whether the current PU is predicted to use affine mode. If the current PU is using affine merging mode, it will obtain the first block encoded in affine mode from the valid neighbor reconstructed blocks. Figure 5 As shown, the selection order of candidate blocks can be from left, top, top right, bottom left to top left (e.g., C0→B0→B1→C1→A0). The affine parameters of the first affine-coded block can be used to derive v0 and v1 of the current PU.

[0074] In video standards such as HEVC, the decoding MV of each PU can be downsampled (e.g., at a 16:1 ratio) and stored in a time MV buffer for use in MVP derivation of the next frame. In the case of a 16x16 block, only the MV of the top left 4x4 is stored in the time MV buffer, and the stored MV represents the MV of the entire 16x16 block.

[0075] Affine MVP pruning

[0076] In affine inter-frame and / or affine merge modes, if any MVP list needs to be constructed, affine MVPs can be pruned using MVPs already existing in the list. In affine inter-frame and affine merge using separate lists, other affine MVPs in the list can be used to prune the affine MVP to be added. However, in affine merge using a unified list, the affine MVP to be added can be pruned not only using existing affine MVPs in the list but also using other non-affine MVPs in the list.

[0077] When an affine MVP is pruned using only affine MVPs, different affine mechanisms can be applied. Furthermore, the affine MVP pruning techniques described herein can be applied to the construction of separate affine candidate lists, such as affine AMVP MVP candidate lists and / or affine merged candidate lists.

[0078] In some embodiments, pruning can be performed through full pruning. For example, each affine MVP to be added can be pruned using all existing affine MVPs.

[0079] In some embodiments, Figure 5 Affine MVPs in groups B, C, and A can first be pruned within their own groups. Then, the surviving candidates from two of the three groups are pruned by the MVPs surviving from the third group. For example, an affine MVP from C1 is pruned by an affine MVP from C0, an affine MVP from B1 is pruned by an affine MVP from B0, an MVP from group B is pruned by an MVP from A0, and an MVP from group C is pruned by an MVP from A0. In some embodiments, in group A, only position A0 can be searched.

[0080] In some embodiments, the affine MVP to be added can only be pruned from one of the MVPs in the list. For example, the affine MVP to be added can only be pruned from the first affine MVP in the MVP list, the second affine MVP in the MVP list, or the last affine MVP in the MVP list.

[0081] In some embodiments, an affine MVP can only be pruned by other MVPs in its own group. For example, an affine MVP from group B can only be pruned by other MVPs in group B, an affine MVP from group C can only be pruned by other MVPs in group C, and an affine MVP from group A can only be pruned by other MVPs in group A.

[0082] In some embodiments, the affine MVP of C0 is first added to the list, then the affine MVP of B0 is added, and B0 is used to compare with C0. Then the affine MVP of B1 is added, and B1 is used to compare with B0. Then the affine MVP of C1 is added, and C1 is used to compare with C0. Then the affine MVP of A0 is added, and A0 is used to compare with C0 and B0.

[0083] Pruning can be performed by comparing the similarity of control point MVs. In some embodiments, the affine MVP to be added will be pruned if the MV difference is less than a threshold. In other embodiments, the affine MVP to be added will only be pruned if the control point MVs of the two MVPs are the same.

[0084] When an affine MVP is pruned using affine MVPs from a unified merge list and other non-affine MVPs, different affine mechanisms can be applied.

[0085] In some embodiments (e.g., in addition to the pruning techniques described above), the affine MVP generated from a neighboring block can be pruned by other non-affine MVPs. In some embodiments, the affine MVP to be added will be pruned if: the control point MV of the affine MVP to be added is the same as or similar to the MV of an existing non-affine MVP; or the difference between the MVs of the two control points of the affine MVP in a four-parameter affine mode is less than a threshold; or the difference between the MVs of the three control points of the affine MVP in a six-parameter affine mode is less than a threshold.

[0086] In some embodiments, if the existing non-affine MVP is from the ATMVP pattern and the reference MV is from the affine block, the affine MVP to be added will be trimmed based on the control points of the affine MVP to be added and the similarity between the top left, top right, and bottom left MVs using the ATMVP pattern.

[0087] Affine MV derivation

[0088] For an affine pattern with four parameters, the MV of each sub-block can be derived using the following Equation 3:

[0089]

[0090] Where w is the distance between two control points V0 and V1.

[0091] Figure 6 For affine blocks using a four-parameter affine pattern, the exemplary positions of control points v0 and v1, 602 and 604, are shown individually by icon 600. Figure 6 In the example, w can be equal to the width of the current block because the positions of the two control points 602 and 604 are at the top left and top right corners of the block. In such an example, the divisor w in Equation 3 can be simply achieved by displacement, since the width of the block can be a power of two.

[0092] The inventors have discovered and recognized that storing the MV of a control point can introduce errors when the control point is being referenced. For example, this is because the MV of the control point represents movement at a corner of a sub-block, rather than at the center of the sub-block. The inventors have developed a technical improvement to existing affine patterns to address these problems. In some embodiments, the technique includes storing the MV at the center of the sub-block (e.g., instead of...). Figure 6 (Displayed in the corner). Figure 7 Icon 700 shows an example where the MVs of two control points v0 and v1 are stored in the center of the upper left and upper right sub-blocks 702 and 704, as shown in positions 706 and 708 respectively.

[0093] The inventors further discovered and recognized that, in some cases, storing the MV at the center of a sub-block can lead to the need for complex division when deriving the MV of a sub-block from an affine block. For example, complex division might be required because the distance between the MVs of two control points might become a number that is not a power of two. Figure 7 In the example, the block is a 16x16 block, so the distance between positions 706 and 708 is 12, not a power of two.

[0094] In some embodiments, the positions of the control points can be configured to be adjusted such that the distance between the control points (e.g., w) is a power of two. In some embodiments, the positions of one or more control points can be moved to change the distance between the control points to a power of two. For example, the position of the second control point can be changed such that the distance between the two control points is equal to a power of two. Figure 8 Icon 800, according to some embodiments, illustrates an example of changing the position of control points in four-parameter affine mode. Illustration 800 uses sub-blocks 802 and 804, which include control points v0 and v1 at center positions 806 and 808 respectively. Figure 7 Comparing this to example 700, in this example, control point v0 is in the upper left sub-block 802, while control point v1 has moved to sub-block 804, which is one sub-block to the left of the upper right sub-block. As shown in the figure, the block is a 16x16 block, so the distance between v0 and v1 is 8 (or 23).

[0095] In some embodiments, the MV of a sub-block can be determined using the new / corrected position of the control point. For example, in four-parameter affine mode, to derive the MV of a sub-block whose control point position has changed, the following Equation 4 can be used (e.g.) Figure 8 As shown):

[0096]

[0097] Where v1′=(v1x ′,v 1y ') is the center MV of the new control point; and

[0098] W′ is the new distance between two control points, and is restricted to a power of two.

[0099] In some embodiments, the MV of each sub-block can be derived using the following Equation 5 in the six-parameter affine mode:

[0100]

[0101] Where W is the width of the block; and

[0102] V0, V1, and V2 are three control points at the corners of the current block, and are restricted to powers of two.

[0103] Figure 9 Icon 900, according to some embodiments, shows the position of the control points of the affine block using the six-parameter affine mode. As shown, control points V0, V1, and V2 are located at positions 902, 904, and 906, respectively.

[0104] In some embodiments, the positions of one or more control points in the six-parameter affine pattern can be moved away from corner positions (e.g., Figure 9 (As shown). Taking the example illustrated here, when the MV is being referenced and / or needs to be determined, overly complex divisions can be avoided. In these cases, the position of the control point can be moved so that the MV better represents the sub-block. Figure 10 Icon 1000, according to some embodiments, shows an example of the position of control points after modification using a six-parameter affine mode. Illustration 1000 uses sub-blocks 1002, 1004, and 1006, which include control points v0, v1, and v2 at center positions 1008, 1010, and 1012, respectively. Figure 9 Using example 900 as a comparison, in this example, control point v0 is in the upper left sub-block 1002, while control point v1 moves to sub-block 1004 (which is one sub-block to the left of the upper right sub-block), and control point v2 moves to sub-block 1006 (which is one sub-block above the lower left sub-block). As shown in the figure, the block is a 16x16 block, so the distance between v0 and v1 is 8 (or 23).

[0105] After moving the position of the control point (e.g.) Figure 10 As shown), the MV of the sub-block can be derived using the following equation 6:

[0106]

[0107] Where v1′=(v 1x ′,v1y v') and v2' = (v') and v2' = (v') 2x ′,v 2y ') is the center MV of the new control point;

[0108] W′ is the new distance between the upper left control point and the upper right control point; and

[0109] H′ is the new distance between the upper left control point and the lower left control point.

[0110] Reference index reorder for AMVP mode

[0111] In some embodiments, the reference frame index can be recorded based on the candidate reference frames in the merge list.

[0112] Since merge candidates with smaller indices are more often selected, the reference frames among those candidates are also more often selected in AMVP mode.

[0113] In some embodiments, in AMVP mode, only the first candidate reference frame in the merge list is recorded as index 0. For example, if the first candidate reference frame in the merge list is frame 5, and its original reference frame index is 2, then in AMVP mode, the reference index of frame 5 will be changed to 0; for reference frames whose original reference index is 0, their reference index will be changed to 1; and for reference frames whose original reference index is 1, their reference index will be recorded as 2. In some embodiments, reference indices equal to 0 and 2 can be interchanged. For example, if the reference frame with reference index 2 is frame 5, and the reference frame with reference index 0 is frame 3, then the reordered reference indices for frame 5 and frame 3 will be 0 and 2, respectively.

[0114] In some embodiments, reference frame indices for more than one reference frame can be recorded. The reordered index depends on the merge index of the merge candidates. For example, if a codec in AMVP mode uses the first three merge candidates to record reference frame indices, and the reference frame for the first merge candidate is frame 5, the reference frame index for frame 5 is frame 5, and the reference frame index for frame 3 is frame 3, then the reference frame index for frame 5 can become 0, and the reference frame index for frame 3 can become 1. In some embodiments, the reference frame indices for frames 5 and 3 are recorded first, and the remaining reference indices are filled in by the remaining reference frames, starting with the reference frame with the smallest original reference index. For example, if the original reference indices 0, 1, 2, and 3 correspond to frames 1, 4, 3, and 5 respectively, then the recorded reference indexes will become reference indices 0, 1, 2, and 3 corresponding to frames 5, 3, 1, and 4 respectively. In some embodiments, reference frames can be swapped. For example, if the original reference indices 0, 1, 2, and 3 correspond to frames 1, 4, 3, and 5 respectively, then the record reference indices will become reference indices 0, 1, 2, and 3 corresponding to frames 5, 3, 4, and 1 respectively.

[0115] Partial coefficient coding

[0116] The inventors have discovered and recognized that when most non-zero coefficients appear in a small region, using the diagonal scanning method for the coefficients may introduce redundant repetitions. For example, if like... Figure 11 Since most non-zero coefficients appear in the upper region of the current block, encoding coefficients in the lower region would be wasteful. The technique described here first divides the coefficient block into several regions, and depending on the position of the last coefficient, a flag can be sent to indicate whether only some coefficients need to be scanned. In some embodiments, a coefficient block is divided into four regions. In one example, these four regions are of equal size. If the last coefficient is located in the upper right region, a flag can be sent to indicate whether only the coefficients in the upper left and upper right regions need to be scanned. Figure 12An example is shown where only a portion of the coefficients are scanned. In other embodiments, a coefficient block is divided into four equally sized regions. If the last coefficient is located within the lower left region, a flag can be sent to indicate whether only the coefficients in the upper left and lower left regions need to be scanned. In some embodiments, if the last coefficient is located within the lower right region, all coefficients in the current block need to be scanned, eliminating the need for an additional flag to indicate whether only the coefficients in the upper left and lower left regions need to be scanned. In some embodiments, the current block can be divided into four regions, where these four regions can be of different sizes. The upper left region can be smaller than the other three parts, or the upper left plus upper right region can be smaller than the lower left plus lower right region, and / or the upper left plus lower left region can be smaller than the upper right plus lower right region.

[0117] In some embodiments, the partial coefficient encoding / decoding is applied only to intra-prediction blocks. In some embodiments, the partial coefficient encoding / decoding is applied only to intra-prediction blocks, and only to certain intra-modes or certain intra-MPM modes. In some embodiments, the partial coefficient encoding / decoding is applied only to inter-prediction blocks. In some embodiments, the partial coefficient encoding / decoding is applied only to non-square blocks. In some embodiments, the partial coefficient encoding / decoding is applied only to square blocks. In some embodiments, the partial coefficient encoding / decoding is applied only to blocks whose size / area / width / height is less than a certain threshold. In some embodiments, the partial coefficient encoding is applied only to blocks whose size / area / width / height is greater than a certain threshold. In some embodiments, the partial coefficient encoding is applied only to certain transformations, such as DCT-5 / 8 and / or DST-7. Figure 11 Icon 1100 shows an example of redundancy coefficient encoding according to some embodiments. Figure 12 Icon 1200 shows an example in some embodiments where only a portion of the coefficients in a coefficient block are scanned.

[0118] Control point position

[0119] In affine motion models, the MV of each sample or sub-block is derived from the MV of the control points. For example... Figure 15 As shown in Figure 1500, the control point can be located at one of the four corners of the current block. Figure 15 In the current block, it is divided into several sub-blocks. For the top-left sub-block, its control point is located at... Figure 16The diagram shows the TL (top left) position of sub-block 1600. For the top right sub-block, its control point is located at the TR (top right) position. For the bottom left sub-block, its control point is located at the BL (bottom left) position. For the bottom right sub-block, its control point is located at the BR (bottom right) position.

[0120] The control point of the current block can be changed. In one embodiment, as shown below... Figure 17 As shown by icon 1700, the control point can be located at the center of the four corner sub-blocks (e.g., Figure 16 Position C).

[0121] For example, in some embodiments, when the center position of the four corner sub-blocks (e.g. Figure 16 The position C) is used as a control point to derive the MV of the affine control points, then the difference between MV and V is... B2_x –V B0’_x This will be multiplied by a scaling factor (as described here: (posCurPU_Y – posB2_Y) / RefPU) B The scaling factor is calculated using the formulas `_width` and `(posCurPU_Y – posB2_Y) / (posB3_X – posB2_X)`. If the denominator of the scaling factor is a power of 2, simple multiplication and shift can be applied. However, if the denominator is not a power of 2, a divider is required, which occupies a significant amount of silicon wafer area. To reduce implementation costs, a table, multipliers, and shifters can be used instead of a divider. Since the denominator of the scaling factor is the control point distance of the reference block, its value is smaller than the size of the CTU and is related to the possible CU size. Therefore, the possible values ​​of the scaling factor denominator are finite. For example, this value could be a power of 2 minus 4 (e.g., 4, 12, 28, 60, 124). For these denominators (e.g., D), a table of beta values ​​can be defined beforehand. N / D can be replaced by N*K>>L, where N is the numerator of the scaling factor, L can be a fixed value, and K is related to D and can be derived from the table. For example, for a fixed value of L, the value of K depends on D and can be derived using Table 1 or Table 2 below. For example, the value of L can be 10, the value of K is equal to (256, 85, 37, 17, 8), and D is equal to (4, 12, 28, 60, 124).

[0122] Table 1

[0123]

[0124]

[0125] Table 2

[0126]

[0127] In another embodiment, if the distance between two affine control points is not a power of 2, a scaling factor can be multiplied to simulate the MVD of a distance of 2 between the two control points. For example, if the distance between two control points MV is 124, then the MVD of these two control points can be multiplied by 128 / 124 to simulate the MVD of a distance of 128 between the two control points. Therefore, for a denominator (e.g., D) equal to (4, 12, 28, 60, 124, or 252), scaling factors (8 / 4, 16 / 12, 32 / 28, 64 / 60, 128 / 124, 256 / 252) can be used. The scaling factor M / D can be replaced by P>>L, where M equals D plus the sub-block size (e.g., 4), and L can be a fixed value, while P is related to D and can be derived from the table. For example, for a fixed value of L, the value of P depends on D and can be derived using Table 3 or Table 4 below.

[0128] Table 3

[0129]

[0130] Table 4

[0131]

[0132] In another embodiment, the scaling factor can be replaced by the MV scaling method and / or merging candidate derivations used in AMVP. The MV scaling module can be reused. For example, the motion vector mv can be scaled in the following ways:

[0133] tx = (16384 + (Abs(td) >> 1)) / td

[0134] distScaleFactor=Clip3(-4096,4095,(tb*tx+32)>>6)

[0135] mv=Clip3(-32768,32767,Sign(distScaleFactor*mvLX)*

[0136] ((Abs(distScaleFactor*mvLX)+127)>>8))

[0137] Where td equals the denominator and tb equals the numerator. For example, as described here, tb can be (posCurPU_Y – posB2_Y) and td can be (posB3_X – posB2_X).

[0138] In another embodiment, when the distance between corner MVs (i.e., the distance between control point MVs, i.e., posB3_X – posB2_X) is not a power of 2 (that is, when the center positions of the four corner sub-blocks (e.g., ...) are not powers of 2) Figure 16If position C) is used as a control point, a divider is needed to derive the affine candidate. In one embodiment, the reference MV selection described above can be used. The reference MV selected for affine candidate derivation has a distance of a power of 2. The reference MV selection depends on the position of the reference block, the aspect ratio of the reference block, the position of the current CU, the aspect ratio of the current CU, or a combination of the above information. For example, the MV of the upper left sub-block of the reference block can be selected as the first reference MV, the second MV can be the MV of the sub-block to the right of the first MV with a distance of width / 2 (if width / 2 is a power of 2), and the third MV can be the MV of the sub-block below the first MV with a distance of height / 2 (if height / 2 is a power of 2). For example, if the position of the first MV is (pos1_x, pos1_y), the position of the second MV can be (pos1_x + width / 2, pos1_y), and the position of the third MV can be (pos1_x, pos1_y + height / 2). In another example, if the bottom-left MV of the reference block is taken as the first MV, then the second MV can be the MV of the child block to the right of the first MV at a distance of width / 2 (if width / 2 is a power of 2), and the third MV can be the MV of the child block above the first MV at a distance of height / 2 (if height / 2 is a power of 2). For example, if the position of the first MV is (pos1_x, pos1_y), the position of the second MV can be (pos1_x + width / 2, pos1_y), and the position of the third MV can be (pos1_x, pos1_y – height / 2). In another example, if the bottom-right MV of the reference block is taken as the first MV, then the second MV can be the MV of the child block to the left of the first MV at a distance of width / 2 (if width / 2 is a power of 2), and the third MV can be the MV of the child block above the first MV at a distance of height / 2 (if height / 2 is a power of 2). For example, if the position of the first MV is (pos1_x, pos1_y), the position of the second MV could be (pos1_x – width / 2, pos1_y), and the position of the third MV could be (pos1_x, pos1_y – height / 2). In another example, if the upper right MV of the reference block is used as the first MV, then the second MV could be the MV of the child block to the left of the first MV at a distance of width / 2 (if width / 2 is a power of 2), and the third MV could be the MV of the child block below the first MV at a distance of height / 2 (if height / 2 is a power of 2). For example, if the position of the first MV is (pos1_x, pos1_y), the position of the second MV could be (pos1_x – width / 2, pos1_y), and the position of the third MV could be (pos1_x, pos1_y + height / 2). Note that if the width (or height) is not a power of 2, then the largest power of 2 that is smaller than the width (or height) can be used.For example, if the position of the first MV is (pos1_x, pos1_y), the position of the second MV can be (pos1_x–(1<<floor(log2(width))), pos1_y). Generally speaking, if the MV at the center position of the sub-block is used for affine candidate derivation, the distance between the two selected reference MVs can be (1<<floor(log2(width-4)) or (1<<floor(log2(height-4)). It should be noted that in some cases, only the first MV and the second MV, or the first MV and the third MV will be used for affine candidate derivation.

[0139] In the affine candidate derivation of one embodiment, the MV position of the MVP / MVD signaling and / or the MV position of the control point can be repositioned to the first MV, the second MV, and / or the third MV. The distance between the first MV and the second MV and / or the distance between the first MV and the third MV can correspond to the width and the height. For example, as Figure 18A shown in FIG. 1800, the first MV can be the MV at the center position of the upper left sub-block, the second MV can be the MV to the right of the first MV with a distance of width / 2, and the third MV can be the MV below the first MV with a distance of height / 2, as Figure 18A shown. To derive the affine candidates (such as AMVP or merge candidates) of the current block, the MVs at the positions of the first, second, and / or third MVs can derive CP0, CP1, CP2 as Figure 18A shown. For the AMVP mode, the MVDs at these positions will be signaled. From the MVs of these two or three control points, the MV of each sub-block can be derived and stored. The stored MVs can be used in motion compensation and as reference MVs for neighboring blocks. For example, the MVs adjacent to CP0, CP1, and / or CP2 can be used to derive the MVs of the control points of CP0, CP1, and / or CP2 of the current block. In one embodiment, for the corner-derived affine candidates, the positions of the neighboring blocks are changed to the positions of the blocks close to the new control point positions. For example, the MVs of these blocks A0, A1, A2, B0, B1, C0, C1 in Figure 18A can be used.

[0140] The corner-derived affine candidates can also be derived using the temporal combined MV, or filled into the candidate list when the candidate list is not full. The temporal candidate can be at the position (top-left_pos1_x+width / 2, top-left_pos1_y+height / 2). For example Figure 18A the position of col-1 in Figure 18A can also be used, and the positions of col-2, col-3, col-4 in

[0141] In some embodiments, a CTU / CTU-row linebuffer constraint can also be applied. For example, when the reference block is in the aforementioned CTU / CTU-row, the lower left MV of the reference block is used as the first MV, and the second MV can be the MV to the right of the first MV at a distance of width / 2 (if width / 2 is a power of 2). Alternatively, if the lower right MV of the reference block is used as the first MV, the second MV can be the MV to the left of the first MV at a distance of width / 2 (if width / 2 is a power of 2). For example, a linear buffer constraint such as... Figure 18B The MVs of CP2 and CP3. With these two MVs, the four-parameter affine candidate can be derived.

[0142] It should be noted that, according to some embodiments, the derived control point MV or affine parameter can be used in inter-frame mode coding such as MVP or merge mode coding such as affine merge candidate.

[0143] AMVP (Average MVP) set derivation and candidate list filling.

[0144] In some embodiments, affine candidates can be generated from an affine candidate list. In some embodiments, a set of candidates (such as inheritance candidates and / or structural candidates) can be added to the candidate list. If the candidate list is not full after adding the first set of candidates, a second set of candidates (such as zero motion vector candidates, additional derivation candidates, and / or time candidates) can be used to fill the candidate list. Figure 19According to some embodiments of the technology described herein, a computerized method 1900 for generating a candidate list is illustrated in flowchart form. Method 1900 can be performed by various means, such as an encoder and / or decoder. In step 1902, the means decides to use affine prediction to encode and decode the current block of video data. In step 1904, the means generates a first set of candidates for the current block, which may include inherited candidates and / or structural candidates. For example, the means may determine one or more inherited candidates, each inheriting the affine model of an associated neighboring block. As another example, the means may derive one or more structural candidates, each derived based on multiple associated neighboring blocks of the current block. In some embodiments, if the affine prediction is associated with a four-parameter affine pattern, each candidate in the candidate list includes two motion vectors: a first motion vector about a first control point and a second motion vector about a second control point. If the affine prediction is associated with a six-parameter affine pattern, then each candidate in the candidate list includes three motion vectors: a first motion vector about the first control point, a second motion vector about the second control point, and a third motion vector about the third control point.

[0145] In step 1906, after the first set of candidates is generated, the device determines whether the candidate list is full. If the candidate list is full, the method proceeds to step 1910 and stops. If the candidate list is not full, the method proceeds to step 1908 and fills the candidate list by generating a second set of candidates. In some embodiments, the device may generate one or more zero motion vector candidates; generate one or more additional derived candidates based on multiple associated neighboring blocks of the current block; and / or generate a temporal motion vector candidate based on a temporally fused image. When the candidate list is full in step 1908, the method proceeds to step 1910 and ends.

[0146] Referring to step 1904, in some embodiments, inherited affine candidates that inherit a neighboring block's affine pattern can be added, such that the control points MV of the aforementioned candidates are derived from the same neighboring block. For example, in some embodiments, structure candidates can also be generated, which contain MVs from different neighboring blocks. In some embodiments, inherited candidates can be added to the candidate list before structure candidates.

[0147] In some embodiments, to determine structural candidates, the technique can determine a first available motion vector (MV) from one or more determined sets of neighboring blocks. See also... Figure 5For example, the technique can find the first available MV for V0 of A0, A1, and A2, which can be represented by MVA. In another example, the technique can find the first available MV for V1 of B0 and B1, which can be represented by MVB. In yet another example, the technique can find the first available MV for V2 of C0 and C1 (… Figure 5 (Not shown in the image) Find the first available MV, which can be represented by MVC. In some embodiments, the technique can find MVs pointing to the same target reference image in two lists. For example, the technique can find MVs in the order {B0-L0, B0-L1, B1-L0, B1-L1}.

[0148] In some embodiments, one or more orders can be used to determine the first available MV. As described herein, for example, using the first available MV in group-A, the search order can be {A0, A1, A2} or {A0, A2, A1}. In some embodiments, a limited set can also be used; for example, only A0 can be used in group-A. For group-B, the search order can be {B0, B1} or {B1, B0}. In some embodiments, for example, only a limited set such as B0 can be used in group-B. In some embodiments, only B1 can be used in group-B. For group-C, for example, the search order can be {C0, C1} or {C1, C0}. In some embodiments, for example, only C0 can be used in group-C. In some embodiments, only C1 can be used in group-C. In some embodiments, only A0, B1, and C1 can be used. In some embodiments, only A0 can be used in group-A, the search order in group-B is {B1, B0}, and the search order in group-C is {C1, C0}. For group-C, if the reference image and target reference image in lists LIST_0 and LIST_1 are not the same, MV scaling can be applied. If an MV is available in group-C, MV scaling is not applied to groups-B and group-C. Otherwise, MV scaling can be applied to groups-B and group-C.

[0149] In some embodiments, the technique may add only one structural candidate to the candidate list (e.g., after adding one or more inheritance candidates). For example, for a four-parameter affine pattern, the technique may insert candidate {MV} into the candidate list. A ,MV B For example, for a six-parameter affine pattern, the technique can insert candidate {MV} into the candidate list. A ,MV B ,MVc}.

[0150] Referring to step 1906, in some embodiments, if the candidate list is not full after all affine inheritance candidates and structure candidates have been added, one or more zero motion vector sets (zero MVsets) can be added. In some embodiments, for affine merging candidates, in a P-slice, two or three zero MVs (MVx=0, MVy=0 and / or MVz=0) can be used as two or three control point MVs and added to the candidate list. For example, in AMVP mode (or non-merging mode), zero MV sets whose reference image is equal to the current target reference image are added until the candidate list is full. In merging mode, for example, in a P-slice, the zero MV sets of list LIST_0 are used; in a B-slice, the zero MV sets of lists LIST_0 and LIST_1 can be used.

[0151] In some embodiments, zero motion vectors (MVs) with different reference indices can also be inserted into the candidate list. The apparatus can determine a number of reference images associated with the candidate list and generate that number of zero motion vector candidates, such that each generated zero motion vector candidate has a separate reference index (refIdx), which is selected from 0 to (the number of reference images - 1). For example, if list_0 has 3 reference images, then three groups of zero MVs with reference indices refIdx equal to 0, 1, and 2 can be added to the candidate list. If the candidate list is not yet full, the group of zero MVs with reference index refIdx equal to 0 will be added until the candidate list is full.

[0152] In some embodiments, if the candidate list is not full after all affine inheritance candidates and structure candidates have been added, general AMVP candidates or general merge candidates can be added to the candidate list. Two or three control points MV can be filled with general AMVP candidate MV or general merge candidate MV.

[0153] In some embodiments, if the candidate list is not full after all affine inheritance candidates and structural candidates have been added, then the available MV in group-A (A0, A1, and A2) (such as the first) is... A The MV available in group-B (B0 and B1) (as in the first group) B , and MV available in group-C (C0 and C1) (as in the first). CThese can then be used to fill the affine candidate list. In some examples, affine candidates with the same MV can be used. In some embodiments, two or three control point MVs (such as four- or six-parameter affine patterns) can all be the same MV from group-A, group-B, or group-C. For example, in a six-parameter affine pattern, the three candidate control point MVs can all include the same MV at each control point, such as {MV... A ,MV A ,MV A}、{MV B ,MV B ,MV B}、{MV C ,MV C ,MV C}. Another example is in a four-parameter affine pattern, where each of the two candidate control points MV can include the same MV, such as {MV...} A ,MV A}、{MV B ,MV B}、{MV C ,MV C}

[0154] In some embodiments, in a six-parameter affine pattern, the filling order used to generate candidates can be one or more of the following: A→B→C, A→C→B, C→B→A, C→A→B, B→A→C, or B→C→A. In some embodiments, in a four-parameter affine pattern, the filling order used to generate candidates can be one or more of the following: A→B, A→C, C→B, C→A, B→A, or B→C. When used to generate candidates to add to the candidate list, the available MVs in group-A, group-B, and group-C can be checked in a predetermined order. For example, the order described herein could be {A0,A1,A2}, {B0,B1}, and {C0,C1}. In some embodiments described herein, if after generating candidates with the same MV at each control point, and the candidate list is not yet full, affine candidates with zero MV can be used to fill the candidate list.

[0155] In some embodiments, if the candidate list is still not full (after adding affine inheritance candidates and / or structure candidates), candidates can be generated to fill the candidate list using the following method: MV A With MV B weighted average, MV A With MV C Weighted average, or MV B With MV C The weighted average.

[0156] In the derivation of affine MVP, corner spatial neighboring (MV) can be used to derive the control point MV (e.g., Figure 5 (MV in Group-A, Group-B, and Group-C). In some embodiments, the techniques described herein can use time-parallel MV as an affine candidate derivation, which can use MV... Col To indicate. For example... Figure 14 As shown in Figure 1400, a scaled time merge MV can be used at position Col-H (e.g., the lower right sub-block of the time merge block adjacent to the current block) and / or position Col-CBR (e.g., the lower right sub-block of the center of the time merge block of the current block). In some embodiments, the MV of Col-H can be selected first if available. If the MV of Col-H is unavailable, the MV of Col-CBR can be used. In some embodiments, the position of Col-H can be considered. For example, if the position of Col-H is not within the merge CTU row (e.g., the position of Col-H is in the top CU row of a CTU, or Col-CBR is in the bottom CU row of a CTU), or if Col-H is unavailable, the MV of Col-CBR can be used. In some embodiments, a time merge MV (from Col-H or Col-CBR) can be used for each control point.

[0157] In some embodiments, it is possible to use MV A MV B With MV C , and / or MV A MV B With MV C Temporal candidates are filled in before the weighted average. For example, in a six-parameter affine pattern, the following four three-control points MV can be added to the list: {MV Col ,MV Col ,MV Col}、{MV A ,MV A ,MV A}、{MV B ,MV B ,MV B}、{MV C ,MV C ,MV C}. In another example, in a four-parameter affine pattern, the following four two-control points MV can be added to fill the list: {MV Col ,MV Col}、{MV A ,MVA}、{MV B ,MV B}、{MV C ,MV C}

[0158] In some embodiments, it is possible to use MV A MV B With MV C , and / or MV A MV B With MV C The weighted average is then used to fill in the time MV (temporal MV). In some embodiments, a pruning procedure can be applied to the MV filling. For example, if the filling order is MV... Col MV A MV B With MV C MV Col It will be added to the list first. The technology may include adding MV candidates (such as MV) later. A MV B With MV C ) and MV Col Comparisons are made. If a later-added MV is identical to one of the already filled MVs, then the later-added MV is not added to the candidate list. In some cases, a to-be-added filled MV can be compared with a group of MVs already in the list; if all control point MVs are identical, then the to-be-added filled MV is not added to the candidate list.

[0159] In some embodiments, time-based MVs can be used to replace the MVs in groups-A, B, and C. For example, if the first available MV in group-A is unavailable, a time-based MV can be used to replace the MVA. In some embodiments, time-based MVs can be used to fill MVP groups when the MVP group candidate list is not yet full.

[0160] In some embodiments, if MVA is unavailable, but {MV B ,MV C ,MV Col There are two usable music videos (such as MV). B With MV C Then, the MVA can be derived from these two available MVs.

[0161] In some embodiments, in Affine Inter-Frame Mode (AMVP mode), if the affine model of a neighboring block is not the same as the current target, this neighboring block is not used to generate inherited affine candidates; instead, only neighboring blocks encoded with the same affine model are used. In some embodiments, affine candidates generated from neighboring blocks encoded with the same affine model are inserted first, followed by affine candidates generated from neighboring blocks encoded with different affine models. In another embodiment, affine candidates generated from neighboring blocks encoded with different affine models are inserted first, followed by affine candidates generated from neighboring blocks encoded with the same affine model.

[0162] In some embodiments, during the affine merging candidate derivation, a six-parameter affine model generated from a neighboring block encoded by a six-parameter affine model is inserted first, followed by a four-parameter affine model generated from a neighboring block encoded by a four-parameter affine model. In another embodiment, a four-parameter affine model generated from a neighboring block encoded by a four-parameter affine model is inserted first, followed by a six-parameter affine model generated from a neighboring block encoded by a six-parameter affine model.

[0163] As described herein, various techniques are available for generating candidates in a candidate list, including adding zero MV, additional derived MV, and / or temporal MV. In some embodiments, the techniques can be used to populate the list in a predetermined order (e.g., after adding inheritance and structural candidates). For example, the techniques can add one or more additional derived candidates (which have the same MV for each control point), a temporal candidate, and a zero MV candidate. For example, this can be achieved by adding {MV... C ,MV C ,MV C}、{MV B ,MV B ,MV B}、{MV A ,MV A ,MV A}、{MV Col ,MV Col ,MV Col}、{MV Zero ,MV Zero ,MV Zero} to fill in the list. Another example is by adding {MV Col ,MV Col ,MV Col}、{MV C ,MV C ,MV C}、{MV B ,MV B ,MV B}、{MV A ,MVA ,MV A}、{MV Zero ,MV Zero ,MV Zero Use} to fill in the list.

[0164] Affine candidate derivation – all same model

[0165] The affine inheritance candidate derivation reuses the affine model and / or affine parameters of the neighboring block. In one example, if the neighboring block uses a four-parameter affine model, a candidate for a four-parameter affine model is derived; if the neighboring block uses a six-parameter affine model, a candidate for a six-parameter affine model is derived.

[0166] When deriving affine AMVP candidates, several neighboring blocks are searched. In one embodiment, for each neighboring block, the MVs in the same target reference list are checked. If a reference list is not the target reference list, then the MVs in that reference list will not be used.

[0167] In one embodiment of affine candidate derivation (such as in affine merge candidate derivation or affine inter-frame mode MVP candidate derivation), a four-parameter affine model is always used. In one example, the four-parameter affine model is always used in inherited affine candidate derivation, regardless of the affine model used by the reference block. If the reference block uses a six-parameter affine model, only two reference control points MV are used to derive the four-parameter affine candidate. In one example, if the reference block is the block above the current block (or the upper right block, or the upper left block), the bottom-left and bottom-right control points MV are used. If the reference block is the block to the left of the current block (or the lower left block, or the upper left block), the upper-right and bottom-right control points MV are used. In another example, the four-parameter affine model is always used in corner-derivation / structure affine candidate derivation. Four-parameter affine candidates can be derived as follows: the (first) available MVs in group-A (A0, A1, and A2) and the (first) available MVs in group-B (B0 and B1); the (first) available MVs in group-A (A0, A1, and A2) and the (first) available MVs in group-C (C0 and C1); or the (first) available MVs in group-B (B0 and B1) and the (first) available MVs in group-C (C0 and C1). For affine candidates, only two-control-point MVs are constructed.

[0168] In another embodiment of inherited affine candidate derivation, a six-parameter affine model is always used. In another embodiment, a six-parameter affine model is always used in corner-derived affine candidate derivation. In another embodiment, a six-parameter affine model is always used in both inherited and corner-derived affine candidate derivations. In one example, a six-parameter affine model is always used in inherited affine candidate derivation, regardless of the affine model used by the reference block. If the reference block uses a four-parameter affine model, only two reference control points MV are used to derive the two control points of the current block (e.g., top-left and top-right control points), while the third control point can be derived from the reference block or the two reference control points MV of the current block. In affine merge mode derivation, affine candidates are marked as six-parameter affine candidates. In another example, for all affine-coded blocks, if a four-parameter affine model is used, the third control point MV (e.g., bottom-left or bottom-right MV) is derived and stored, and the affine-coded block is also marked as a six-parameter model. When performing inherited affine model derivation, a six-parameter affine model is used (for affine candidate derivation, three corner control points MV are used directly, instead of deriving the third control point MV online). The six-parameter affine model is used during sub-block MV derivation and motion compensation of affine-coded blocks. In one embodiment of the affine inter-frame mode (sometimes also called affine AMVP mode), if a four-parameter affine model is selected, a list of two-control-point MVP pairings is generated. One of these MVP pairings is selected, and the two MVDs are encoded and the two control points MVs are constructed by pairing this MVP pairing with the MVD. Then, the four-parameter affine model is used to generate the third control point MV of the current block. This block is then marked as a six-parameter affine model or treated as a six-parameter affine model. The six-parameter affine model is used during sub-block MV derivation and motion compensation of affine-coded blocks. In another example, the six-parameter affine model is always used in corner-derivation / structure affine candidates. Only three-control-point MV structure affine candidates can be added to the candidate list. A six-parameter affine candidate can be derived as follows: the (first) available control point MV in group-A (A0, A1, and A2) and the (first) available control point MV in group-B (B0 and B1), or the (first) available control point MV in group-A (A0, A1, and A2) and the (first) available control point MV in group-C (C0 and C1). If one or two of the control point MVs are unavailable, the missing control point MVs are derived from the available control point MVs using a four-parameter affine model. For example, the following candidates can be used: {top left, top right}, {top left, bottom left}, {top right, bottom right}, {bottom left, bottom right}, {top left, bottom right}, {top right, bottom left}. However, for such two control point candidates, a four-parameter affine model is used to generate the missing control point MVs under the six-parameter affine model.For example, for the candidate {top left, top right}, the bottom left MV will be generated. A six-parameter {top left, top right, -bottom left} candidate will be used instead of a four-parameter {top left, top right} candidate. In one embodiment, all affine merge candidates are labeled as six-parameter affine candidates.

[0169] By using the techniques described herein, in some embodiments, it may be unnecessary to store information about whether the current block or reference block is a four-parameter or six-parameter affine model. In some embodiments, it is also unnecessary to use a buffer to store the aforementioned information. If a six-parameter affine model is always used, then only the six-parameter affine model will be used during the sub-block MV derivation and motion compensation process of the affine-coded block.

[0170] BT-split context modeling

[0171] In JEM, the BT-partitioning context model depends on 2*QT-depth + 1*BT-depth. According to the technique described here, the area of ​​nearby blocks can be used for content modeling. If the combined area of ​​the left and upper CUs is less than the area of ​​the current CU, the first content is used. If the area of ​​either the left or upper CU is less than the area of ​​the current CU, the second content is used. If the combined area of ​​the left and upper CUs is equal to or greater than the area of ​​the current CU, the third content is used.

[0172] In another example, if the combined area of ​​the left and top CUs is equal to or less than the area of ​​the current CU, the first content is used. If the area of ​​either the left or top CU alone is equal to or less than the area of ​​the current CU, the second content is used. If the combined area of ​​the left and top CUs is greater than the area of ​​the current CU, the third content is used.

[0173] Any of the above techniques can be implemented in the encoder and / or decoder. For example, it can be implemented in an encoder ( Figure 2 In the MV derivation module of encoder 200, and / or a decoder ( Figure 3 Any of the above-described techniques may be implemented in the MV derivation module of the decoder 300. Alternatively, any of the above-described techniques may be implemented as a circuit coupled to the MV derivation module of the encoder and / or the MV derivation module of the decoder to provide the information required by the MV derivation module.

[0174] The technical operations based on the principles described herein can be implemented in any suitable manner. The processing and decision blocks in the flowcharts above represent steps and actions within algorithms that perform these different processes. Algorithms derived from these processes can be implemented using software integrated into and directing the operation of one or more single-purpose or multi-purpose processors, can be implemented using functional equivalent circuitry such as Digital Signal Processing (DSP) circuitry or Application-Specific Integrated Circuit (ASIC), or can be implemented in any other suitable manner. It should be understood that the flowcharts included herein do not describe the syntax or operations of any particular circuit or any particular programming language or type of programming language. Rather, the flowcharts above illustrate functional information that would be used by those skilled in the art to manufacture circuits or implement computer software algorithms to perform the processing of a particular apparatus performing the type of technology described herein. It should also be understood that, unless otherwise stated herein, the specific order of steps and / or actions described in each flowchart is merely illustrative of the algorithm, which can be implemented based on the principles described herein and can be varied in implementation and embodiment.

[0175] Therefore, in some embodiments, the techniques described herein can be implemented in computer-executable instructions implemented as software, including application software, system software, firmware, middleware, embedded code, or any other suitable type of computer code. These computer-executable instructions can be written and compiled into executable machine language code or intermediate code that executes on a framework or virtual machine using any of a variety of suitable programming languages ​​and / or programming or scripting tools.

[0176] When the techniques described herein are implemented as computer-executable instructions, these instructions can be implemented in any suitable manner, including as a number of functional facilities, each providing one or more operations to perform the execution of algorithmic operations according to these techniques. A “functional facility” (however exemplified) is a structural component of a computer system that, when integrated and executed by one or more computers, causes one or more computers to perform a specific operational role. A functional facility can be part of an entire software element. For example, a functional facility can be implemented as a function of a processor, or as a discrete processor, or as any other suitable processor. If the techniques described herein are implemented as multiple functional facilities, each functional facility can be implemented in its own manner; it is not necessary for all of these functional facilities to be implemented in the same manner. Furthermore, these functional facilities can be executed in parallel and / or serially, and, where appropriate, shared memory on the computer can be used to pass information to each other, by using a message passing protocol, or by executing these functional facilities on the computer in any other suitable manner.

[0177] Generally, functional facilities include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Typically, the functionality of functional facilities can be combined or distributed according to the requirements of the system in which they operate. In some implementations, one or more functional facilities that perform the techniques described herein can together form a complete software package. In alternative embodiments, these functional facilities may be adapted to interact with other unrelated functional facilities and / or processes to implement software application applications.

[0178] This document has described some exemplary functional facilities for performing one or more tasks. However, it should be understood that the described functional facilities and task divisions are merely illustrative of the types of functional facilities used to implement the exemplary techniques described herein, and embodiments are not limited to implementation in any particular number, division, or type of functional facility. In some implementations, all functions may be implemented in a single functional facility. It should also be understood that in some implementations, some of the functional facilities described herein may be implemented together with others or separately (i.e., as a single unit or several separate units), or some functional facilities may not be implemented.

[0179] In some embodiments, computer-executable instructions implementing the technologies described herein (when implemented as one or more functional facilities or in any other way) are encoded on one or more computer-readable media to provide functionality to the media. Computer-readable media include magnetic media such as hard disk drives, optical discs (CDs) or digital versatile optical discs (DVDs), persistent or non-persistent solid-state memory (such as flash memory, magnetic random access memory, etc.), or any other suitable storage medium. Such computer-readable media can be implemented in any suitable manner. As used herein, “computer-readable medium” (also referred to as “computer-readable storage medium”) means a tangible storage medium. A tangible storage medium is non-transitory and has at least one physical, structural component. In the “computer-readable medium” as used herein, at least one physical structural component has at least one physical property that can be altered in some way during the creation of the medium having the implemented information, during the recording of information thereon, or in any other process of encoding the medium having the information. For example, during the recording process, the magnetization state of a portion of the specific physical structure of the computer-readable medium can be changed.

[0180] Furthermore, some of the techniques described above involve storing information (e.g., data and / or instructions) in a certain way for use in the behavior of these techniques. In some implementations of these techniques—for example, when the techniques are implemented as computer-executable instructions—information may be encoded on a computer-readable storage medium. If the particular structure described herein is an advantageous format for storing that information, these structures, when encoded on the storage medium, can be used for the organization of information transmitted to a specific physical entity. These advantageous structures can then provide functionality to the storage medium by influencing the operation of one or more processors interacting with that information; for example, by improving the efficiency of computer operations performed by the processor.

[0181] In some, but not all, implementations of the technology as computer-executable instructions, these instructions can be executed on one or more suitable computing devices operating on any suitable computer system, or one or more computing devices (or one or more processors of one or more computing devices) can be programmed to execute the computer-executable instructions. A computing device or processor can be programmed to execute instructions when the instructions are stored in a manner accessible to the computing device or processor, such as in data storage (e.g., an on-chip cache or instruction register, a computer-readable storage medium accessible via a bus, a computer-readable storage medium accessible via one or more networks and accessible by the device / processor, etc.). Functional facilities including these computer-executable instructions can be integrated into and direct the operation of a single multipurpose programmable digital computing device, a coordinated system of two or more multipurpose computing devices sharing processing power and jointly executing the technologies described herein, a coordinated system of a single computing device or a computing device dedicated to executing the technologies described herein (located in the same place or geographically distributed), one or more field-programmable gate arrays (FPGAs) implementing the technologies described herein, or any other suitable system.

[0182] A computing device may include at least one processor, a network adapter, and a computer-readable storage medium. For example, a computing device may be a desktop computer or laptop, a personal digital assistant (PDA), a smartphone, a server, or any other suitable computing device. The network adapter may be any suitable hardware and / or software enabling the computing device to communicate wired and / or wirelessly with any other suitable computing device over any suitable computing network. A computer network may include wireless access points, switches, routers, gateways, and / or other network devices, as well as any suitable wired and / or wireless communication media or media for exchanging data between two or more computers, including the Internet. The computer-readable medium may be adapted to store data to be processed and / or instructions to be executed by the processor. The processor can process the data and execute the instructions. Data and instructions may be stored on the computer-readable storage medium.

[0183] Computing devices may also have one or more components and peripherals, including input devices and output devices. In addition to those already described, these devices may also be used to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for outputting visual presentations, and speakers or other sound-generating devices for outputting auditory presentations. Examples of input devices that can be used for a user interface include keyboards and pointing devices such as mice, touchpads, and digitizing tablets. As another example, a computing device may receive input information via speech recognition or other audible formats.

[0184] Embodiments implementing these techniques in the form of circuits and / or computer-executable instructions have been described. It should be understood that some embodiments may be in the form of a method, of which at least one example is provided. Actions performed as part of a method may be ordered in any suitable manner. Therefore, these embodiments can be interpreted as performing actions in an order different from that shown, and thus, even with respect to actions performed sequentially as shown in the illustrated embodiments, they can be interpreted as including actions performed simultaneously.

[0185] The various aspects of the embodiments described above can be used individually, in combination, or even in various arrangements not specifically discussed in the above embodiments, and their application is therefore not limited to the specific details and arrangements of the components in the foregoing description or drawings. For example, an aspect described in one embodiment can be combined in any way with aspects described in other embodiments.

[0186] The use of sequential terms such as "first," "second," and "third" in the claims of a patent application to modify the elements of the claims does not in itself imply any priority, sequence, or temporal order of the execution of actions or methods in the order of the elements of the claims; rather, it is merely used to mark and distinguish one element of the claims with the same name from another element with the same name (used only as sequential terms), thereby distinguishing the elements of the claims.

[0187] Furthermore, the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The terms “including,” “contains,” “has,” “includes,” “involves,” and their variations, as used herein, refer to the items listed thereafter, their equivalents, and any additional items.

[0188] As used herein, the term "exemplary" means as an example, instance, or illustration. Therefore, unless otherwise stated, any embodiments, implementations, processes, features, etc., described herein should be understood as illustrative examples and not as preferred or advantageous examples.

[0189] Having described several aspects of at least one embodiment, it should be understood that various changes, modifications, and improvements will not be difficult for those skilled in the art. Such changes, modifications, and improvements will be part of the invention and within the spirit and scope of the principles described herein. Therefore, the above description and figures are by way of example only.

Claims

1. A video data encoding and decoding method, comprising: The decision is made to use affine prediction to encode the current block of video data; Generate a list of candidates for the current block, including: When one or more inheritance candidates are determined to be available, each of these candidates is determined to inherit an affine model of an associated neighboring block; and When it is determined that one or more structure candidates are available, the one or more structure candidates are derived, wherein each structure candidate is derived based on multiple associated neighboring blocks of the current block; If it is determined that the candidate list is not yet full, the current candidate list is populated by generating other candidates from the candidate list, including; When one or more additional derivation candidates are determined to be available, the one or more additional derivation candidates are generated based on multiple associated neighboring blocks of the current block, and the generated one or more additional derivation candidates are added to the candidate list immediately following the one or more structural candidates. This involves determining available motion vectors based on a first set of neighboring blocks, generating a first candidate including the first available motion vector, and generating the first candidate for each control point of the first candidate. If it is determined that the candidate list is not yet full, a second available motion vector is determined based on a second set of neighboring blocks, which is different from the first set of neighboring blocks, and a second available motion vector is generated. The second candidate is generated for each control point of the second candidate; it is determined that the candidate list is not yet full, and a third available motion vector is determined based on a third group of nearby blocks, which is different from the first group and the second group of nearby blocks. A third candidate including the third available motion vector is generated for each control point of the third candidate, wherein the first group of nearby blocks is the first sub-block of the current block adjacent to the associated lower left control point; the second group of nearby blocks is the second sub-block of the current block adjacent to the associated upper right control point; and the third group of nearby blocks is the third sub-block of the current block adjacent to the associated upper left control point. When the candidate list is not yet full, one or more temporal motion vector candidates are generated based on the temporally merged image, and the generated one or more temporal motion vector candidates are added to the candidate list immediately after the one or more additional derived candidates; and When the candidate list is not full after filling it with one or more additional derived candidates, the candidate list is filled by deriving a zero motion vector candidate that needs to be added to the candidate list, wherein the zero motion vector candidate is immediately after the one or more additional derived candidates in the candidate list.

2. The video data encoding and decoding method according to claim 1, characterized in that: If the affine prediction is associated with a four-parameter affine pattern, then each candidate in the candidate list includes a first motion vector about a first control point and a second motion vector about a second control point; and If the affine prediction is associated with a six-parameter affine pattern, then each candidate in the candidate list includes the first and second motion vectors, and a third motion vector with respect to the third control point.

3. The video data encoding and decoding method according to claim 1, characterized in that, The candidate motion vector for this time period is generated as follows: Motion vectors are available from a block of the time-paneled image, which is the lower-right sub-block of the time-paneled block adjacent to the current block in the time-paneled image; and Generate the time motion vector candidate that includes the motion vector, and generate it for each control point of the time motion vector candidate.

4. The video data encoding and decoding method according to claim 1, characterized in that, Further includes: Motion vectors are unavailable from a block of the time-paneled image that is the lower-right sub-block of the time-paneled block adjacent to the current block in the time-paneled image; and From the lower right sub-block of the time-space block, a time motion vector candidate including the motion vector is generated, and this is done for each control point of the time motion vector candidate.

5. The video data encoding and decoding method according to claim 1, characterized in that, Candidates for generating this time motion vector include: Based on the temporal merging image, the candidate motion vectors for that temporal motion vector are determined; and The motion vector is scaled before generating the time motion vector candidate.

6. The video data encoding and decoding method according to claim 1, characterized in that, Generating one or more zero motion vector candidates includes: Determine a number of reference images to be associated with the candidate list; and Generate the number of zero motion vector candidates, each of which includes a separate reference index selected from a set of integers from zero to the number minus one.

7. The video data encoding and decoding method according to claim 6, characterized in that, Further includes: It has been determined that the candidate list is not yet full; Generate a remaining number of zero motion vector candidates to be added to the candidate list, wherein the reference index of each of the remaining number of zero motion vector candidates is zero.

8. The video data encoding and decoding method according to claim 1, characterized in that, Other candidates that generate this candidate list include those that use the same motion vector for each control point.

9. An apparatus for video data encoding and decoding, the apparatus comprising a processor for communicating with memory, the processor executing instructions stored in the memory such that the processor: The decision is made to use affine prediction to encode the current block of video data; Generate a list of candidates for the current block, including: When one or more inheritance candidates are determined to be available, each of these candidates is determined to inherit an affine model of an associated neighboring block; and When it is determined that one or more structure candidates are available, the one or more structure candidates are derived, wherein each structure candidate is derived based on multiple associated neighboring blocks of the current block; If it is determined that the candidate list is not yet full, the current candidate list is populated by generating other candidates from the candidate list, including; When one or more additional derivation candidates are determined to be available, the one or more additional derivation candidates are generated based on multiple associated neighboring blocks of the current block, and the generated one or more additional derivation candidates are added to the candidate list immediately following the one or more structural candidates. This involves determining available motion vectors based on a first set of neighboring blocks, generating a first candidate including the first available motion vector, and generating the first candidate for each control point of the first candidate. If it is determined that the candidate list is not yet full, a second available motion vector is determined based on a second set of neighboring blocks, which is different from the first set of neighboring blocks, and a second available motion vector is generated. The second candidate is generated for each control point of the second candidate; it is determined that the candidate list is not yet full, and a third available motion vector is determined based on a third group of nearby blocks, which is different from the first group and the second group of nearby blocks. A third candidate including the third available motion vector is generated for each control point of the third candidate, wherein the first group of nearby blocks is the first sub-block of the current block adjacent to the associated lower left control point; the second group of nearby blocks is the second sub-block of the current block adjacent to the associated upper right control point; and the third group of nearby blocks is the third sub-block of the current block adjacent to the associated upper left control point. When the candidate list is not yet full, one or more temporal motion vector candidates are generated based on the temporally merged image, and the generated one or more temporal motion vector candidates are added to the candidate list immediately after the one or more additional derived candidates; and When the candidate list is not full after filling it with one or more additional derived candidates, the candidate list is filled by deriving a zero motion vector candidate that needs to be added to the candidate list, wherein the zero motion vector candidate is immediately after the one or more additional derived candidates in the candidate list.

10. The video data encoding and decoding apparatus according to claim 9, characterized in that, The candidate motion vector for this time period is generated as follows: Motion vectors are available from a block of the time-paneled image, which is the lower-right sub-block of the time-paneled block adjacent to the current block in the time-paneled image; and Generate the time motion vector candidate that includes the motion vector, and generate it for each control point of the time motion vector candidate.

11. The video data encoding and decoding apparatus according to claim 9, characterized in that, Further includes: Motion vectors are unavailable from a block of the time-paneled image that is the lower-right sub-block of the time-paneled block adjacent to the current block in the time-paneled image; and From the lower right sub-block of the time-space block, a time motion vector candidate including the motion vector is generated, and this is done for each control point of the time motion vector candidate.

12. A non-transitory computer-readable storage medium for storing processor-executable instructions, which, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the following actions: The decision is made to use affine prediction to encode the current block of video data; Generate a list of candidates for the current block, including: When one or more inheritance candidates are determined to be available, each of these candidates is determined to inherit an affine model of an associated neighboring block; and When it is determined that one or more structure candidates are available, the one or more structure candidates are derived, wherein each structure candidate is derived based on multiple associated neighboring blocks of the current block; If it is determined that the candidate list is not yet full, the current candidate list is populated by generating other candidates from the candidate list, including; When one or more additional derivation candidates are determined to be available, the one or more additional derivation candidates are generated based on multiple associated neighboring blocks of the current block, and the generated one or more additional derivation candidates are added to the candidate list immediately following the one or more structural candidates. This involves determining available motion vectors based on a first set of neighboring blocks, generating a first candidate including the first available motion vector, and generating the first candidate for each control point of the first candidate. If it is determined that the candidate list is not yet full, a second available motion vector is determined based on a second set of neighboring blocks, which is different from the first set of neighboring blocks, and a second available motion vector is generated. The second candidate is generated for each control point of the second candidate; it is determined that the candidate list is not yet full, and a third available motion vector is determined based on a third group of nearby blocks, which is different from the first group and the second group of nearby blocks. A third candidate including the third available motion vector is generated for each control point of the third candidate, wherein the first group of nearby blocks is the first sub-block of the current block adjacent to the associated lower left control point; the second group of nearby blocks is the second sub-block of the current block adjacent to the associated upper right control point; and the third group of nearby blocks is the third sub-block of the current block adjacent to the associated upper left control point. When the candidate list is not yet full, one or more temporal motion vector candidates are generated based on the temporally merged image, and the generated one or more temporal motion vector candidates are added to the candidate list immediately after the one or more additional derived candidates; and When the candidate list is not full after filling it with one or more additional derived candidates, the candidate list is filled by deriving a zero motion vector candidate that needs to be added to the candidate list, wherein the zero motion vector candidate is immediately after the one or more additional derived candidates in the candidate list.

Images