Video coding method and apparatus

Abstract

The present disclosure provides a video coding method and related apparatus. The video coding method includes receiving input data associated with a current block; determining one or more neighboring motion vectors from one or more non-adjacent affine coded neighborhoods of the current block; determining a control point motion vector based on the one or more neighboring motion vectors, wherein if a target neighboring block associated with a target neighboring motion vector is outside of an available area, a derived control point motion vector is generated to replace the target neighboring motion vector; generating an affine merge list or an affine advanced motion vector prediction list, wherein a non-adjacent affine candidate uses motion information to generate a non-adjacent affine predictor according to the control point motion vector; and encoding or decoding the current block using a motion candidate selected from the affine merge list or the affine advanced motion vector prediction list. The video coding method and related apparatus of the present disclosure reduce storage requirements.

Description

[Technical Field]

[0001] This invention relates to video encoding and decoding using motion estimation and motion compensation. More specifically, this invention relates to complexity reduction in systems using affine modes with non-adjacent candidates. [Background Technology]

[0002] Unless otherwise stated herein, the methods described in this section are not prior art to the claims listed below, and are not acknowledged as prior art by virtue of their inclusion in this section.

[0003] Universal Video Coding (VVC) is the latest international video coding standard (MPEG) developed by the Joint Video Experts Group (JVET) of the ITU-T Video Coding and Decoding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group. This standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology—Coded representation of immersive media—Part 3: Versatile video coding, published in February 2021. VVC was developed by adding more coding and decoding tools to improve coding and decoding efficiency and handle various types of video sources, including 3D video signals, based on its predecessor, HEVC (High Efficiency Video Coding).

[0004] Figure 1AAn exemplary adaptive inter / intra video codec system incorporating loop processing is described. For intra-frame prediction, prediction data is derived from previously encoded video data in the current frame (hereinafter also referred to as a picture). For inter-frame prediction 112, motion estimation (ME) is performed on the encoder side, and motion compensation (MC) is performed based on the result of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects either intra-frame prediction 110 or inter-frame prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The residual from transform and quantization is then encoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. Then, the bitstream associated with the transform coefficients is packaged together with side information (e.g., motion and decoding modes associated with intra-frame and inter-frame prediction) and other information (e.g., parameters associated with the loop filter applied to the underlying image area). The side information associated with intra-frame prediction 110, inter-frame prediction 112, and loop filter 130 is provided to the entropy encoder 122, such as... Figure 1A As shown. When using inter-frame prediction mode, one or more reference images must also be reconstructed at the encoder end. Therefore, the transformed and quantized residuals are processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residuals. Then, at reconstruction (REC) 128, the residuals are added back to the prediction data 136 to reconstruct the video data. The reconstructed video data can be stored in the reference image buffer 134 and used to predict other frames.

[0005] like Figure 1A As shown, the input video data undergoes a series of processing steps in the encoding system. Due to these processing steps, the reconstructed video data from REC 128 may suffer various forms of degradation. Therefore, loop filters 130 are frequently applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve video quality. For example, deblocking filters (DF), Sample Adaptive Offset (SAO), and Adaptive Loop Filters (ALF) can be used. It may be necessary to incorporate loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, loop filter information is also provided to entropy encoder 122 for incorporation into the bitstream. Figure 1A In the process, the loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in the reference image buffer 134. Figure 1A The system described herein is intended to illustrate an exemplary architecture of a typical video encoder. It may correspond to a High Efficiency Video Codec (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0006] like Figure 1B As shown, apart from transform 118 and quantization 120, the decoder can use similar or identical function blocks to the encoder, since the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and the required encoding / decoding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not require mode search. Instead, the decoder only needs to generate intra-frame predictions based on the intra-frame prediction information received from entropy decoder 140. Furthermore, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from entropy decoder 140 without motion estimation.

[0007] Similar to HEVC, VVC partitions the input image into non-overlapping square block regions called CTUs (Code-Decoder Tree Units). Each CTU can be further divided into one or more smaller Code-Decoder Units (CUs). The resulting CU partitions can be squares or rectangles. Furthermore, VVC divides the CTUs into Prediction Units (PUs), which serve as units for applying prediction processing, such as inter-frame prediction, intra-frame prediction, etc.

[0008] The VVC standard incorporates various new codec tools to further improve encoding and decoding efficiency based on the HEVC standard. Among these new codec tools, some relevant to this invention are summarized below.

[0009] Affine Optical Flow

[0010] When the codec unit (CU) is encoded and decoded using affine patterns, the CU is divided into 4x4 sub-blocks, and for each sub-block, a motion vector is derived based on the affine model, and motion compensation is performed to generate the corresponding predictor. The reason for using a 4x4 block as a sub-block instead of using other smaller sizes is to achieve a good trade-off between the computational complexity of motion compensation and the efficiency of encoding and decoding. To improve encoding and decoding efficiency, JVET-N0236 (J. Luo et al., “CE2-related: Prediction refinement with optical flow for affinemode”, ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Experts Group (JVET), 14th Meeting: Geneva, Switzerland, March 19-27, 2019, Document: JVET-N0236), and JVET-N0261 (K. Zhang et al., “CE2-1.1: Interweaved Prediction for Affine Motion Compensation”, ITU-T SG 16WP3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Experts Group (JVET), 14th Meeting: Geneva, Switzerland, March 19-27, 2019, Document: JVET-N0261), and JVET-N0262 (H. Huang et al., “CE9-related: Disabling DMVR for non equal weight BPWA”, Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC29 / WG 11, 14th meeting: Geneva, Switzerland, 19-27 March 2019, document: JVET-N0262) discloses several methods.

[0011] In JVET-N0236, to achieve finer granularity of motion compensation, the contribution proposes a method for refining sub-block-based affine motion compensation predictions using optical flow. After performing sub-block-based affine motion compensation, the brightness prediction samples are refined by adding the difference derived from the optical flow equation. The proposed Prediction Refinement with Optical Flow (PROF) is described in the following four steps: Step 1), perform sub-block-based affine motion compensation to generate sub-block predictions I(i,j). Step 2), compute the spatial gradient g of the sub-block prediction at each sample location using a 3-tap filter [-1,0,1]. x (i,j) and g y (i,j).

[0012] g x (i,j)=I(i+1,j)-I(i-1,j), and

[0013] g y (i,j)=I(i,j+1)-I(i,j-1).

[0014] Sub-block prediction extends by one pixel on each side for gradient calculation. To reduce memory bandwidth and complexity, pixels on the extended boundaries are copied from the nearest integer pixel position in the reference image. This avoids additional interpolation in the filled regions. In step 3), brightness prediction refinement is calculated using the optical flow equation.

[0015] ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j)

[0016] Where Δv(i,j) is the pixel MV (denoted by v(i,j)) calculated for sample position (i,j) and the sub-block MV (denoted by v) of sub-block 220 of block 210 to which pixel (i,j) belongs. SB The difference between (212) is as follows: Figure 2 As shown. In Figure 2 In the diagram, sub-block 222 corresponds to the motion vector v. SB(212) points to the reference sub-block 220. Reference sub-block 222 represents the reference sub-block generated by the translational motion of block 220. Reference sub-block 224 corresponds to the reference sub-block with PROF. The motion vector of each pixel is refined by Δv(i,j). For example, the sub-block MV v is based on the sub-block modified by Δv(i,j)216. SB (212) Derive the thinned motion vector v(i,j)214 of the top left pixel of sub-block 220.

[0017] Since the affine model parameters and the pixel position relative to the sub-block center do not change between sub-blocks, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let x and y be the horizontal and vertical offsets from the pixel position to the sub-block center, Δv(x,y) can be derived from the following equation.

[0018]

[0019] For a 4-parameter affine model, the parameters c and e can be derived using the following equation:

[0020]

[0021] For a 6-parameter affine model, the parameters c, d, e, and f can be derived using the following equation:

[0022]

[0023] Where (v 0x ,v 0y ),(v 1x ,v 1y ),(v 2x ,v 2y ) are the controlpoint motion vectors for the top left, top right, and bottom left corners, and w and h are the width and height of the CU. Step 4), finally, lumaprediction refinement is added to the sub-block prediction I(i,j). The final prediction I' is generated as shown in the following equation.

[0024] I′(i,j)=I(i,j)+ΔI(i,j).

[0025] In JVET-N0261, Figure 3Another sub-block-based affine mode interleaving prediction method is proposed. Through interleaving prediction, codec block 310 is divided into sub-blocks (320 and 322) with two different partitioning modes. Then, two auxiliary predictions (P0 330 and P1 332) are generated using affine motion compensation based on the two partitioning modes. The final prediction 340 is calculated as a weighted sum of the two auxiliary predictions (330 and 332). To avoid motion compensation for 2×H or W×2 block sizes, for... Figure 4 The two partitioning modes shown have an interweaved prediction mode that is only applied to regions with a sub-block size of 4×4.

[0026] According to the method disclosed in JVET-N0261, affine motion compensation based on 2x2 sub-blocks is only applied to uni-prediction of luminance samples, and the 2x2 sub-block motion field is only used for motion compensation. Storage of the motion vector field used for motion prediction, etc., remains based on 4x4 sub-blocks. If bandwidth constraints are applied, affine motion compensation based on 2x2 sub-blocks is disabled when the affine motion parameters do not meet certain criteria.

[0027] In JVET-N0273 (H. Huang et al., “CE9-related: Disabling DMVR for nonequalweight BPWA”, Joint Video Experts Group (JVET) 1 / SC 29 / WG 11, 14th Meeting: Geneva, Switzerland, March 19-27, 2019, Document: JVET-N0262), 2x2 subblock-based affine motion compensation is only applicable to unidirectional prediction of luminance samples, and the 2x2 subblock motion field is used only for motion compensation. If bandwidth constraints are applied, 2x2 subblock-based affine motion compensation will be disabled when the affine motion parameters do not meet specific criteria.

[0028] Affine Model

[0029] Motion occurring across images along the time axis can be described by many different models. Suppose A(x, y) is the original pixel at the considered position (x, y), and A′(x′, y′) is the corresponding pixel at position (x′, y′) in the reference image of the current pixel A(x, y). The affine motion model is described as follows.

[0030] Affine models can describe the rotation and deformation of two-dimensional blocks to transform a square (or rectangle) into a parallelogram. This model can be described as follows:

[0031] x' = a0 + a1x + a2y, and

[0032] y'=b0+b1*x+b2*y. (1)

[0033] The ITU-T13-SG16-C1016 document (Lin et al., “Affine transform prediction for next generation video coding”, ITU-U, Study Group 16, Question Q6 / 16, Document C1016, Geneva, Switzerland, 2015) submitted to ITU-VCEG discloses a four-parameter affine prediction, including an affine merging mode. When an affine moving block is in motion, the block's motion vector field can be described by two control point motion vectors or four parameters as follows, where (vx, vy) represents the motion vector...

[0034]

[0035] An example of a four-parameter affine model is as follows: Figure 5 As shown, the corresponding reference block 520 of the current block 510 is located based on an affine model with two control point motion vectors (i.e., v0 and v1). The transformed block is a rectangular block, and the motion vector field of each point in this moving block can be expressed by the following formula:

[0036]

[0037] or

[0038]

[0039] In the above equation, (v 0x ,v 0y ) is the motion vector of the control point at the top left corner of the block (i.e., v0), (v 1x v 1y Let v1 be the motion vector of another control point at the top right corner of the block. When decoding the MV of the two control points, the MV of each 4x4 block of the block can be determined according to the above equation. In other words, the affine motion model of the block can be specified by the two motion vectors at the two control points. Furthermore, although the top left and top right corners of the block are used as the two control points, other two control points can also be used. An example of determining the motion vector of the current block for each 4x4 sub-block based on the MV of the two control points can be given according to equation (3).

[0040] In ITU-T 13-SG16-C1016, for CUs encoded in inter-frame mode, when the CU size is equal to or greater than 16x16, an affine flag is sent to indicate whether affine inter-frame mode is applied. If the current block (e.g., the current CU) is encoded in affine inter-frame mode, a list of candidate MVP pairs is constructed using adjacent valid reconstructed blocks. Figure 6 The diagram illustrates the neighboring block set used to derive corner-derived affine candidates. For example... Figure 6 As shown, The motion vector corresponding to block V0 at the top left corner of the current block 610 is selected from the motion vectors of neighboring blocks A0 (called the above-left block), A1 (called the inner above-left block), and A2 (called the lower above-left block). The motion vector corresponding to block V1 in the upper right corner of the current block 610 is selected from the motion vectors of neighboring blocks B0 (called the upper block) and B1 (called the upper-right block).

[0041] Document ITU-T13-SG16-C1016 also proposes an affine merge mode. If the current block 710 is a merged encoding / decoding PU, then the five adjacent blocks are checked ( Figure 7 The system uses blocks AO, A1, B0, B1, and B2 to determine whether any of them are encoded or decoded in affine inter-mode or affine merge mode. If so, it sends an affine_flag to indicate whether the current PU is in affine mode. When the current PU is in affine merge mode, it retrieves the first block encoded in affine mode from the valid adjacent reconstructed blocks. The selection order of candidate blocks is from left block (A1), top block (B1), top right block (B0), bottom left block (A0), to top left block (B2). In other words, the search order is A1→B1→B0→A0→B2, as shown below. Figure 7 As shown. The affine parameters of the affine codec block are used to derive v0 and v1 of the current PU. Figure 7 In the example, the neighborhood blocks (A0, A1, B0, B1, and B2) used to construct the control points MV for the affine motion model are referred to in this disclosure as a neighborhood block set.

[0042] In affine motion compensation (MC), the current block is divided into multiple 4x4 sub-blocks. For each sub-block, the center point (2, 2) is used to derive the MV using equation (3) for that sub-block. For this current MC, a 4x4 sub-block translation MC is performed for each sub-block.

[0043] In HEVC, the MV decoded by each PU is downsampled at a ratio of 16:1 and stored in a temporal MV buffer for MVP derivation in subsequent frames. For a 16x16 block, only the top-left 4x4 MV is stored in the temporal MV buffer, and the stored MV represents the MV of the entire 16x16 block.

[0044] Bidirectional optical flow (BIO)

[0045] Bidirectional optical flow (BIO) is a motion estimation / compensation technique disclosed in JCTVC-C204 (E. Alshina et al., Bi-directional optical flow, ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Collaborative Working Group on Video Coding (JCT-VC), 3rd Meeting: Guangzhou, China, October 7-15, 2010, Document: JCTVC-C204) and VCEG-AZ05 (E. Alshina et al., Known tools performance investigation for next generation video coding, ITU-T SG 16 Question 6, Video Coding Experts Group (VCEG), 52nd Meeting: June 19-26, 2015, Warsaw, Poland, Document: VCEG-AZ05). BIO is based on... Figure 8 The assumptions about optical flow and stable motion shown derive sample-level motion refinement, where the current pixel 822 in B-slice (bidirectional prediction slice) 820 is predicted from a pixel in reference image 0 (830) and a pixel in reference image 1 (810). Figure 8 As shown, the current pixel 822 is predicted by pixel B (812) in reference image 1 (810) and pixel A (832) in reference image 0 (830). Figure 8 In the middle, v x and v yThese are pixel displacement vectors in the x and y directions, derived using a bidirectional optical flow (BIO) model. It only applies to true bidirectional prediction blocks, predicted from two reference images corresponding to the previous and next images. In VCEG-AZ05, BIO uses a 5x5 window to derive motion refinement for each sample. Therefore, for an NxN block, motion compensation results and corresponding gradient information from (N+4)x(N+4) blocks are needed to derive sample-based motion refinement for the NxN block. According to VCEG-AZ05, a 6-tap gradient filter and a 6-tap interpolation filter are used to generate the gradient information for BIO. Therefore, the computational complexity of BIO is significantly higher than traditional bidirectional prediction. To further improve the performance of BIO, the following method is proposed.

[0046] In regular bidirectional prediction in HEVC, the predictor is generated using the following equation, where P (0) and P (1) These are the predictors for list 0 and list 1, respectively.

[0047] P Conventional [i, j] = (P (0) [i, j] + P (1) [i, j]+1)>>1

[0048] In JCTVC-C204 and VECG-AZ05, the following equation is used to generate BIO predictors:

[0049] POpticalFlow=(P (0) [i, j] + P (1) [i, j] + v x [i, j](I) x (0) -I x (1) [i, j])+v y [i, j](I) y (0) -I y (1) [i, j])+1)>>1

[0050] In the above equation, I x (0) and I x (1) These represent the x-direction gradients in the predictors of list0 and list1, respectively; I y (0) and I y (1) These represent the y-axis gradients in the predictors of list0 and list1, respectively; v xand v y These represent offsets or displacements in the x and y directions, respectively. x and v y The derivation process is shown below. First, the cost function is defined as diffCost(x, y) to find the optimal value v. x and v y To find the optimal value v x and v y Minimize the cost function diffCost(x, y) using a 5x5 window. x and v y The solutions can be represented by S1, S2, S3, S5 and S6.

[0051]

[0052] The minimum cost function mindiffCost(x,y) can be derived as follows:

[0053]

[0054] By solving equations (3) and (4), v x and v y The solution can be found using the following equation:

[0055]

[0056] in,

[0057]

[0058] In the above equation, This corresponds to the gradient in the x-direction of the pixel at (x, y) in the image of list 0. This corresponds to the gradient in the x-direction of the pixel at (x, y) in the image of list 1. The gradient in the y-direction of the pixel at (x, y) in the image corresponding to list 0. The gradient in the y-direction of the pixel at (x,y) in the image corresponding to list 1.

[0059] In some related technologies, S2 can be ignored, v x and v y The solution can be found as follows:

[0060]

[0061] in,

[0062]

[0063] We can observe that the bit depth required in BIO operations is relatively large, especially when calculating S1, S2, S3, S5, and S6. For example, if the bit depth of pixel values ​​in a video sequence is 10 bits, increasing the bit depth of the gradient through fractional interpolation or gradient filters requires 16 bits to represent a gradient in the x-direction or y-direction. This 16 bits can be further reduced using a gradient shift of 4, thus requiring 12 bits to represent a gradient value. Even if the gradient magnitude can be reduced to 12 bits through gradient shifting, the bit depth required for BIO operations remains significant. A 13-bit x 13-bit multiplier is needed to calculate S1, S2, and S5. Another 13-bit x 17-bit multiplier is needed to obtain S3 and S6. When the window size is large, more than 32 bits are required to represent S1, S2, S3, S5, and S6.

[0064] Recently, a new tool called non-adjacent affine candidate has been proposed for emerging video codec standards. However, this new tool requires storing motion information of large neighborhood blocks. This invention discloses a technique to reduce storage requirements. [Summary of the Invention]

[0065] The following overview is illustrative only and is not intended to be limiting in any way. That is, it is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious techniques described herein. Choices, but not all, of implementations are further described in the detailed description below. Therefore, the following overview is not intended to identify the essential characteristics of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter.

[0066] This invention discloses a video encoding / decoding method, comprising: receiving input data associated with a current block, wherein the input data includes pixel data of the current block to be encoded on the encoder side or encoded data associated with the current block to be decoded on the decoder side; determining one or more neighborhood motion vectors from one or more non-adjacent affine encoding / decoding neighborhoods of the current block; determining control point motion vectors based on the one or more neighborhood motion vectors, wherein if a target neighborhood block associated with a target neighborhood motion vector is outside the available area, a derived control point motion vector is generated to replace the target neighborhood motion vector; generating an affine merging list or an affine advanced motion vector prediction list including one or more neighborhood motion vectors as non-adjacent affine candidates, wherein a non-adjacent affine candidate generates a non-adjacent affine predictor using motion information based on the control point motion vector; and encoding or decoding the current block using motion candidates selected from the affine merging list or the affine advanced motion vector prediction list.

[0067] The present invention also discloses an apparatus for video encoding and decoding, the apparatus comprising one or more electronic circuits or processors for: receiving input data associated with a current block, wherein the input data includes pixel data of the current block to be encoded on the encoder side or encoded data associated with the current block to be decoded on the decoder side; determining one or more neighborhood motion vectors from one or more non-adjacent affine encoding and decoding neighborhoods of the current block; determining control point motion vectors based on the one or more neighborhood motion vectors, wherein if a target neighborhood block associated with a target neighborhood motion vector is outside the available area, a derived control point motion vector is generated to replace the target neighborhood motion vector; generating an affine merging list or an affine advanced motion vector prediction list including one or more neighborhood motion vectors as non-adjacent affine candidates, wherein a non-adjacent affine candidate generates a non-adjacent affine predictor using motion information based on the control point motion vector; and encoding or decoding the current block using motion candidates selected from the affine merging list or the affine advanced motion vector prediction list.

[0068] The present invention also discloses a video encoding / decoding method, the method comprising: receiving input data associated with a current block, wherein the input data includes pixel data of the current block to be encoded on the encoder side or encoded data associated with the current block to be decoded on the decoder side; determining one or more neighborhood motion vectors from one or more non-adjacent affine encoding / decoding neighborhoods of the current block; determining control point motion vectors based on the one or more neighborhood motion vectors, wherein if a target neighborhood block associated with a target neighborhood motion vector is outside the available area, a derived control point motion vector is generated to replace the target neighborhood motion vector; generating a motion candidate list including one or more neighborhood motion vectors as a non-adjacent affine candidate, wherein a non-adjacent affine candidate generates a non-adjacent affine predictor using motion information based on the control point motion vector; and encoding or decoding the current block using a motion candidate selected from the motion candidate list.

[0069] The video encoding / decoding method and related apparatus of the present invention reduce storage requirements. [Attached Image Description]

[0070] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of the present disclosure. The drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. It will be understood that the drawings are not necessarily drawn to scale, as in actual implementations, some components may be shown out of proportion to clearly illustrate the concepts of the present disclosure.

[0071] Figure 1A The system described in the document illustrates an exemplary structure of a typical video encoder.

[0072] Figure 1B The diagram shows Figure 1A The corresponding decoder of the encoder in the code.

[0073] Figure 2 The illustration shows an example of affine motion compensation based on sub-blocks, where the motion vectors of each pixel in the sub-block are derived from the thinned motion vectors.

[0074] Figure 3 An example of interleaved prediction is shown, in which the codec block is divided into sub-blocks with two different partition patterns, and then two auxiliary predictions are generated through affine motion compensation with the two partition patterns.

[0075] Figure 4 An example is shown of avoiding motion compensation for interleaving prediction with block sizes of 2×H or W×2, where for both partitioning modes, interleaving prediction is applied only to regions with sub-block sizes of 4×4.

[0076] Figure 5 An example of a four-parameter affine model is illustrated, showing the current block and the reference block.

[0077] Figure 6 An example of affine candidate derivation of inheritance is shown, where the current block inherits the affine model of the neighboring block by inheriting the control point MV of the neighboring block as the control point MV of the current block.

[0078] Figure 7 An example of the constructed affine candidate derivation is shown, where neighborhood blocks (A0, A1, B0, B1, and B2) are used to construct control points MV for the affine motion model.

[0079] Figure 8 An example of sample-level motion refinement derived from bidirectional optical flow (BIO) based on the assumptions of optical flow and stable motion is illustrated.

[0080] Figure 9A -B illustrates an example for deriving non-neighboring spatial neighborhoods (NSAMs) for affine merging patterns, where the patterns for obtaining non-neighboring spatial neighborhoods are... Figure 9A The diagram shows affine merge candidates for deriving inheritance. Figure 9B The diagram shows affine merge candidates used to derive the construct.

[0081] Figure 10 An example of an affine candidate constructed based on non-neighboring neighborhoods is shown, where motion information from three non-neighboring neighborhoods at positions A, B, and C is used to form a CPMV.

[0082] Figure 11 An exemplary flowchart of a video encoding / decoding system utilizing non-adjacent affine candidates according to an embodiment of the present invention is shown.

Detailed Implementation Methods

[0083] It is readily understood that the components of the invention, as generally described and illustrated in the accompanying drawings, can be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the invention as shown in the figures is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the invention. References throughout this specification to “an embodiment,” “one embodiment,” or similar language mean that a particular feature, structure, or characteristic described in connection with that embodiment may include in at least one embodiment of the invention. Therefore, the phrases “in one embodiment” or “in one embodiment” appearing throughout this specification do not necessarily refer to the same embodiment.

[0084] Furthermore, the described features, structures, or characteristics can be combined in one or more embodiments in any suitable manner. However, those skilled in the art will recognize that the invention can be practiced without one or more specific details, or using other methods, components, etc. In other instances, well-known structures or operations are not shown or depicted in detail to avoid obscuring aspects of the invention. The illustrated embodiments of the invention will be best understood with reference to the accompanying drawings, wherein like parts are represented by like numerals throughout. The following description is intended only as an example and simply illustrates certain selected embodiments of devices and methods consistent with the invention as claimed herein.

[0085] Affine with non-adjacent candidates

[0086] In JVET-Y0153 (Wei Chen et al., “EE2-3.11: Non-adjacent spatial neighbors for affine merge mode”, ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Expert Group (JVET), 25th Meeting, Teleconference, January 12-21, 2022, Document: JVET-Y0153), a technique for using non-adjacent spatial neighbors for affine merge and AMVP modes was proposed. In the non-adjacent spatial neighbors for affine merge mode (NSAM), the pattern of the non-adjacent spatial neighbors is obtained as follows: Figure 9A -B is shown, where Figure 9A Used to derive affine merge candidates for inheritance. Figure 9BThe affine merging candidates used to derive the construction. Similar to the existing non-adjacent regular merging candidates disclosed in JVET-X2025 (Fabrice LeLéannec et al., “Algorithm description of Enhanced Compression Model 3 (ECM 3)”, ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Joint Video Expert Group (JVET), 24th Meeting, Teleconference, October 6-15, 2021, Document: JVET-X2025), the distance between non-adjacent spatial neighbors and the current decompiled block in NSAM is also defined based on the width and height of the current CU.

[0087] Figure 9A Motion information of non-adjacent spatial neighborhoods in -B is used for the current CU ( Figure 9A Block 910 and Figure 9B Block 920 in the VVC generates additional affine merge candidates for inheritance and / or construction. Specifically, for inheritance candidates, the same derivation process for affine merge candidates for inheritance remains unchanged, except that the CPMV is inherited from a non-adjacent spatial neighborhood. In other words, in one example, the CPMV may correspond to an MV inherited based on one or more non-adjacent neighborhood MVs, or in another example, to an MV constructed from one or more non-adjacent neighborhood MVs. In yet another example, the CPMV may correspond to an MV inherited based on one or more non-adjacent neighborhood MVs or an MV constructed from one or more non-adjacent neighborhood MVs. Non-adjacent spatial neighborhoods are checked from nearest to farthest neighbor based on their distance from the current block. At a specific distance, only the first available neighborhood on each side of the current block (e.g., the left and top) (i.e., a neighborhood encoded using affine patterns) is included for candidate derivation of inheritance. Figure 9A As shown by the dashed arrows, the inspection order for the left and upper neighbors is bottom-up and right-left, respectively. For candidate constructs (i.e., "affine candidates of the first class of constructs from non-adjacent neighbors"), as... Figure 9B As shown, first, the positions of a non-adjacent spatial neighborhood to the left and above are independently determined. Then, the position of the upper-left reference can be determined accordingly, which, together with the non-adjacent neighborhoods to the left and above, forms a rectangular virtual block. Then, as... Figure 10 As shown, motion information from three non-adjacent neighborhoods at positions A, B, and C is used to form the CPMV of the top left (A), top right (B), and bottom left (C) corners of the virtual block, which are then projected onto the current CU to generate candidates for the corresponding construction.

[0088] In JVET-Y0153, a large number of CPMVs of neighboring CTUs need to be stored. To reduce buffer requirements for data access, it is recommended to limit the available area of ​​non-adjacent spatial neighboring blocks. If a neighboring block associated with a CPMV is outside the available area, the CPMV is considered unavailable or will be replaced by a exported CPMV, which can be the nearest CPMV around the available area, one or more predefined CPMVs, one or more CPMVs exported by a predefined method, or a stored 4x4 or NxN sub-block MV. In this disclosure, for simplicity, we can refer to the situation where a neighboring block associated with a CPMV is outside the available area as "CPMV outside the available area". In one embodiment, the area is within the current CTU. In this case, a non-adjacent spatial neighboring block to be used can only be inserted into the affine merge list or affine AMVP list, or any CPMV candidate list, if it is within the same CTU as the current block. In another embodiment, the area can be the current CTU and the K decoded CTUs to the left. In another embodiment, the region can be the current CTU, the K decoded CTUs to the left, and the L decoded CTUs above. In another embodiment, the region can be N CTU rows. In another embodiment, the region can be a region with multiple CTUs. In yet another embodiment, the region can be the current CTU, the K decoded CTUs to the left, the L decoded CTUs above, the M decoded CTUs to the upper left, and the P decoded CTUs to the upper right. K, L, M, N, and P can be any integer greater than zero. In another embodiment, the available region can be adaptively determined using a method involving the current CU position.

[0089] The predefined available area can be defined according to the processing unit. In one embodiment, the available area for non-adjacent spatial neighborhood blocks for affine merge candidates (i.e., inherited MVs) based on one or more non-adjacent neighborhood MVs and constructed affine merge candidates (i.e. constructed MVs) derived from one or more non-adjacent neighborhood MVs should be the same.

[0090] In one embodiment, if the non-adjacent spatial neighbor block to be used is invalid (i.e., outside the predefined region), a default motion vector can be applied. For example, the default motion vector can be derived from the top-left block of the valid region, the leftmost block of the valid region, or the topmost block of the valid region, or a translation motion vector can be used directly.

[0091] This constraint can also be extended to non-neighboring neighborhood candidates used in regular merging, MMVD, AMVP, IBC, or any other tool that attempts to reference non-neighboring spatial neighborhood candidates.

[0092] In one embodiment, an available region can be predefined for temporally collocated motion vectors. If a temporally collocated motion vector to be referenced is outside this region, it should not be referenced. In one embodiment, the region can be the current CTU, N decoded CTUs, or M decoded CTU rows. M and N can be any integer greater than zero. The available region for temporally collocated motion vectors can be different from the available region for non-adjacent spatial neighborhood candidates.

[0093] Any of the methods proposed above can be implemented in the encoder and / or decoder. For example, any of the proposed methods can be implemented in the affine inter-frame prediction module of the encoder and / or decoder (e.g., Figure 1A Inter-frame prediction 112 or Figure 1B It can be implemented in MC 152. Alternatively, any of the proposed methods can be implemented as a circuit coupled to the affine inter-frame prediction module of the encoder and / or decoder.

[0094] Figure 11 An exemplary flowchart of a video codec system utilizing non-adjacent affine candidates according to an embodiment of the present invention is illustrated. The steps shown in the flowchart can be implemented as program code executable on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented in hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, in step 1110, input data associated with the current block is received, wherein the input data includes pixel data of the current block to be encoded on the encoder side or encoded data associated with the current block to be decoded on the decoder side. In step 1120, one or more neighborhood MVs (motion vectors) are determined based on one or more non-adjacent affine encoded neighborhoods of the current block. In step 1130, a CPMV (control point motion vector) is determined based on the one or more neighborhood MVs, wherein if a target neighborhood block associated with a target neighborhood MV is outside the available area, a derived CPMV is generated to replace the target neighborhood MV. In step 1140, an affine merge list or affine AMVP (Advanced Motion Vector Prediction) list is generated, which includes one or more neighboring MVs as non-neighboring affine candidates. Each non-neighboring affine candidate generates a non-neighboring affine predictor, using motion information based on the CPMV. In step 1150, the current block is encoded or decoded using a motion candidate selected from the affine merge list or affine AMVP (Advanced Motion Vector Prediction) list.

[0095] The flowchart shown is intended to illustrate an example of video encoding and decoding according to the present invention. Those skilled in the art can modify, rearrange, split, or combine each step to implement the invention without departing from its spirit. Specific syntax and semantics have been used in this disclosure to illustrate examples of implementing embodiments of the invention. Those skilled in the art can implement the invention by substituting equivalent syntax and semantics without departing from its spirit.

[0096] The above description is provided to enable those skilled in the art to practice the invention in the context of specific applications and their requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but is accorded the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details have been illustrated to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced.

[0097] The embodiments of the invention described above can be implemented in various hardware, software code, or combinations thereof. For example, one embodiment of the invention may be one or more circuits integrated into a video compression chip or program code integrated into video compression software to perform the processes described herein. Embodiments of the invention may also be program code to be executed on a digital signal processor (DSP) to perform the processes described herein. The invention may also relate to numerous functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). These processors may be configured to perform specific tasks according to the invention by executing machine-readable software code or firmware code that defines specific methods embodied in the invention. The software code or firmware code may be developed in different programming languages ​​and in different formats or styles. The software code may also be compiled for different target platforms. However, different code formats, styles, and languages ​​of the software code, as well as other ways of configuring the code to perform the tasks according to the invention, do not depart from the spirit and scope of the invention.

[0098] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered illustrative rather than restrictive in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All variations falling within the meaning and scope of the equivalents of the claims should be included within their scope.

Claims

1. A method of video coding, comprising: receiving input data associated with a current block, wherein the input data comprises pixel data of the current block to be coded on an encoder side or coded data associated with the current block to be decoded on a decoder side; determining one or more neighbor motion vectors from one or more non-affine coded neighbors of the current block; determining a control point motion vector based on the one or more neighbor motion vectors, wherein a derived control point motion vector is generated to replace a target neighbor motion vector if a target neighbor block associated with the target neighbor motion vector is outside an available region; generating an affine merge list or an affine advanced motion vector prediction list including the one or more neighbor motion vectors as a non-affine affine candidate, wherein the non-affine affine candidate generates a non-affine predictor using motion information according to the control point motion vector; and coding or decoding the current block using a motion candidate selected from the affine merge list or the affine advanced motion vector prediction list.

2. The method of video coding of claim 1, wherein the available region corresponds to a current CTU enclosing the current block, K decoded CTUs to the left of the current block, L decoded CTUs above the current block, or a combination thereof, and wherein K and L are positive integers.

3. The method of video coding of claim 1, wherein the derived control point motion vector corresponds to a control point motion vector in the available region closest to the current block.

4. The method of video coding of claim 1, wherein the derived control point motion vector corresponds to one or more predefined control point motion vectors.

5. The video coding method of claim 1, wherein, the derived control point motion vector is derived according to a predefined method.

6. The video coding method of claim 1, wherein, the derived control point motion vector is derived according to stored 4x4 or NxN subblock motion vectors, wherein N is a positive integer.

7. The method of claim 1, wherein, the control point motion vector corresponds to an inherited motion vector based on the one or more neighbor motion vectors.

8. The video coding method of claim 1, wherein, the control point motion vector corresponds to a constructed motion vector derived from the one or more neighbor motion vectors.

9. The video coding method of claim 1, wherein, the control point motion vector corresponds to a constructed motion vector derived from the one or more neighbor motion vectors or an inherited motion vector based on the one or more neighbor motion vectors.

10. The method of claim 9, wherein, the available region is the same for the constructed motion vector and the inherited motion vector.

11. An apparatus for video coding, the apparatus comprising one or more electronic circuits or processors arranged to: receiving input data associated with a current block, wherein, receive input data associated with a current block, wherein the input data comprises pixel data of the current block to be coded on an encoder side or coded data associated with the current block to be decoded on a decoder side; determine one or more neighbor motion vectors from one or more non-affine coded neighbors of the current block; determine a control point motion vector based on the one or more neighbor motion vectors, wherein a derived control point motion vector is generated to replace a target neighbor motion vector if a target neighbor block associated with the target neighbor motion vector is outside an available region; generate an affine merge list or an affine advanced motion vector prediction list including the one or more neighbor motion vectors as a non-affine affine candidate, wherein the non-affine affine candidate generates a non-affine predictor using motion information according to the control point motion vector; and code or decode the current block using a motion candidate selected from the affine merge list or the affine advanced motion vector prediction list. generating an affine merge list or an affine advanced motion vector prediction list including the one or more neighbor motion vectors as one non-adjacent affine candidate, wherein the one non-adjacent affine candidate generates a non-adjacent affine predictor using motion information according to the control point motion vectors; and encoding or decoding the current block using a motion candidate selected from the affine merge list or the affine advanced motion vector prediction list.

12. A method of video coding, the method comprising: receiving input data associated with a current block, wherein the input data comprises pixel data of the current block to be encoded at an encoder side or encoded data associated with the current block to be decoded at a decoder side; determining one or more neighbor motion vectors from one or more non-adjacent affine coded neighbors of the current block; determining control point motion vectors based on the one or more neighbor motion vectors, wherein if a target neighbor block associated with a target neighbor motion vector is outside an available region, a derived control point motion vector is generated to replace the target neighbor motion vector; generating a motion candidate list including the one or more neighbor motion vectors as one non-adjacent affine candidate, wherein the one non-adjacent affine candidate generates a non-adjacent affine predictor using motion information according to the control point motion vectors; and encoding or decoding the current block using a motion candidate selected from the motion candidate list.

13. The video coding method of claim 12, wherein, The motion candidate list corresponds to a regular merge candidate list, MMVD, an advanced motion vector prediction list, or an intra block copy.

Images