Method and apparatus for regression-based affine merge mode motion vector derivation in video coding systems

By mixing the control point motion vector and affine parameters with the sub-block motion vectors in adjacent regions in video encoding and decoding, the mixed motion vectors are generated, which solves the problem of high computational complexity based on the regression model, and improves the encoding and decoding efficiency and Reduced storage requirements.

CN119032565APending Publication Date: 2024-11-26MEDIATEK INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380034505.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-22
Filing Date
2023-04-21
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The motion vector field technology based on regression model has high computational complexity in video encoding and decoding, which affects the encoding and decoding efficiency.

Method used

By receiving pixel data or encoded data associated with the current block on the encoder and decoder side, the control point motion vector, affine parameter set or regression model is derived, and mixed with sub-block motion vectors in adjacent regions, to generate Mix motion vectors, insert merge candidate lists for encoding or decoding.

Benefits of technology

It reduces the computational complexity, improves the efficiency of video encoding and decoding, reduces storage requirements, and is suitable for hardware implementation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119032565A_ABST
    Figure CN119032565A_ABST
Patent Text Reader

Abstract

A method and apparatus for video coding using regression MV derivation. According to the method, a CPMV, an affine parameter set, or a regression model for the current block is derived based on a non-adjacent affine coding block. The CPMV, the affine parameter set, or the regression model is mixed with adjacent MVs of sub-blocks within adjacent regions associated with at least one of the CPMV, the affine parameter set, or the regression model to generate at least one mixed MV. A set of CPMVs including the mixed MVs is inserted into the merge candidate list. According to another method, a subset of sub-blocks of a non-adjacent affine codec block is determined. A set of CPMVs or SBMVs is derived based on a subset of sub-blocks of a non-adjacent affine coded block. A derived MV is derived for the current block based on regression processing using an input MV including a set of CPMVs or SBMVs associated with the subset.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references

[0002] This disclosure is part of a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 333,592, filed on April 22, 2022. The contents of the above application are incorporated herein by reference. Technical Field

[0003] The present invention relates to a video codec system that uses a regression model codec tool to generate derived motion vectors. In particular, the present invention relates to a method for reducing the computational complexity associated with regression model-based MV derivation. Background Art

[0004] Versatile video coding (VVC) is the latest international video coding standard jointly developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Video Experts Group (JVET) of the ISO / IEC Moving Picture Experts Group (MPEG). The standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Information technology-Coded representation of immersive media-Part 3:Versatile Video Coding, released in February 2021. VVC improves the coding efficiency by adding more coding tools on the basis of its predecessor HEVC (High Efficiency Video Coding), and can also process various types of video sources, including 3-dimensional (3D) video signals.

[0005] Figure 1AAn exemplary adaptive Inter / Intra video coding system including loop processing is illustrated. For intra prediction, prediction data is derived from previously coded video data in the current picture. For inter prediction 112, motion estimation (ME) is performed on the encoder side and motion compensation (MC) is performed based on the results of ME to provide prediction data derived from other pictures and motion data. Switch 114 selects intra prediction 110 or inter prediction 112 and the selected prediction data is provided to adder 116 to form a prediction error, also called a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The transformed and quantized residual is then encoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with side information (e.g., motion and codec modes associated with intra-frame prediction and inter-frame prediction) and other information (e.g., parameters associated with the loop filter applied to the underlying image area). Side information associated with intra-frame prediction 110, inter-frame prediction 112, and in-loop filter 130 is provided to entropy encoder 122, as shown in FIG. Figure 1A As shown. When using inter-frame prediction mode, one or more reference pictures must also be reconstructed at the encoder end. Therefore, the transformed and quantized residual is processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residual. The residual is then added back to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in a reference picture buffer 134 and used to predict other frames.

[0006] like Figure 1A As shown, the input video data undergoes a series of processes in the encoding system. Due to the series of processes, the reconstructed video data from REC 128 may be subject to various impairments. Therefore, the loop filter 130 is often applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve the video quality. For example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF) may be used. It may be necessary to merge the loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, the loop filter information is also provided to the entropy encoder 122 to be merged into the bitstream. In Figure 1A In , a loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in a reference picture buffer 134 . Figure 1A The system in is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to a High Efficiency Video Codec (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0007] like Figure 1B As shown, the decoder can use similar or identical functional blocks as the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of the entropy encoder 122, the decoder uses an entropy decoder 140 to decode the video bitstream into quantized transform coefficients and required codec information (such as ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 on the decoder side does not need to perform a pattern search. Instead, the decoder only needs to generate an intra-frame prediction based on the intra-frame prediction information received from the entropy decoder 140. In addition, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) 140 based on the inter-frame prediction information received from the entropy decoder without motion estimation.

[0008] According to VVC, similar to HEVC, the input picture is partitioned into non-overlapped square block regions called CTUs (Codec Tree Units). Each CTU can be divided into one or more smaller-sized Codec Units (CUs). The generated CU partitions can be square or rectangular. In addition, VVC divides CTUs into Prediction Units (PUs) as units for applying prediction processing (e.g., inter-frame prediction, intra-frame prediction, etc.).

[0009] Merge Mode

[0010] In order to improve the codec efficiency of motion vector (MV) encoding and decoding in HEVC, HEVC has skip and merge modes. The skip and merge modes obtain motion information from spatially neighboring blocks (spatial candidates) or temporally co-located blocks (temporal candidates). When the PU is in skip or merge mode, the motion information is not encoded and decoded. Instead, only the index of the selected candidate is encoded and decoded. For skip mode, the residual signal is forced to zero and is not encoded and decoded. In HEVC, if a particular block is encoded as skip or merge, the candidate index is signaled to indicate which candidate in the candidate set is used for the merge. Each merged PU reuses the motion information, including the MV, prediction direction, and reference picture index of the selected candidate.

[0011] For the merge mode in the HM-4.0 test model of HEVC, for the current CU 210, up to four spatial MV candidates (i.e., A0, A1, B0, and B1) and one temporal MV candidate (i.e., TBR, or TCTR if TBR is not available) are checked, as shown in Figure 2 As shown. Note that if any of the four spatial MV candidates is not available, position B2 is then used to derive an MV candidate as a replacement. After the derivation process of four spatial MV candidates and one temporal MV candidate, redundancy removal (pruning) is applied to remove any redundant MV candidates. If the number of available MV candidates is less than 5 after redundancy removal (pruning), three types of additional candidates are derived and added to the candidate set (i.e., candidate list). The encoder selects a final candidate from the candidate set in skip or merge mode based on the rate-distortion optimization (RDO) decision and transmits the index to the decoder.

[0012] Although the regression based motion vector field (RMVF) technique has been shown to improve the coding efficiency, the computational complexity associated with the RMVF technique is very high. Therefore, the present invention discloses various methods and devices to reduce the computational complexity associated with the RMVF technique. Summary of the invention

[0013] A method and apparatus for video encoding and decoding using a regression-based motion vector field (RMVF) technique are disclosed. According to the method, pixel data associated with a current block is received at the encoder side or encoded data associated with a current block to be decoded is received at the decoder side. One or more CPMVs (control point motion vectors), affine parameter sets, or regression models of the current block are derived based on non-adjacent affine encoding and decoding blocks. At least one of the one or more CPMVs, at least a portion of the affine parameter set, or at least a portion of the regression model is mixed with adjacent MVs (motion vectors) of sub-blocks in a neighboring area associated with at least one of the one or more CPMVs, the at least part of the affine parameter set, or the at least part of the regression model to generate at least one mixed MV (motion vector). The CPMV set including the at least one mixed MV is inserted into a merge candidate list. The current block is encoded or decoded using information including the merge candidate list.

[0014] In one embodiment, the one or more CPMVs correspond to a lower left CPMV (control point motion vector), an upper left CPMV, and an upper right CPMV. In addition, a lower left mixed MV is generated for the CPMV, an upper left mixed MV is generated for the upper left CPMV, and an upper right mixed MV is generated for the upper right CPMV.

[0015] In one embodiment, the at least one mixed MV is generated using a weighted sum of the at least one of the one or more CPMVs and adjacent MVs. In one embodiment, the weight of the weighted sum depends on the block distance, the block size, or both. In another embodiment, the weight of the weighted sum is predefined.

[0016] According to another method, a subset of sub-blocks of non-adjacent affine codec blocks is determined, wherein the subset of sub-blocks is less than the total number of sub-blocks in the non-adjacent affine codec blocks. A set of CPMVs (control point motion vectors) or SBMVs (sub-block motion vectors) associated with a subset of sub-blocks of a current block is derived based on the subset of sub-blocks of the non-adjacent affine codec blocks. One or more derived MVs (motion vectors) are derived for the current block based on a regression process using an input MV including a set of CPMVs or SBMVs associated with a subset of sub-blocks of the non-adjacent affine codec blocks. The one or more derived MVs are inserted into a merge candidate list. The current block is encoded or decoded using information including the merge candidate list.

[0017] In one embodiment, a set of CPMVs or SBMVs corresponds to a lower left SBMV (sub-block motion vector), an upper left SBMV, and an upper right SBMV. In one embodiment, a derived MV of the current block is determined based on the lower left SBMV, the upper left SBMV, and the upper right SBMV.

[0018] In one embodiment, the input MV also includes SBMVs of neighboring sub-blocks of the current block. A first derived MV of the current block is determined based on a set of CPMVs or SBMVs associated with a subset of sub-blocks of non-adjacent affine codec blocks; a second derived MV of the current block is determined based on the SBMVs of neighboring sub-blocks of the current block; and a final derived MV is generated by mixing the first derived MV and the second derived MV. In one embodiment, the final derived MV corresponds to a weighted sum of the first derived MV and the second derived MV, and the first derived MV has a higher weight than the second derived MV. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1A An exemplary adaptive Inter / Intra video coding system incorporating loop processing is described.

[0020] Figure 1B Shows Figure 1A The corresponding decoder of the encoder in .

[0021] Figure 2 An example of deriving merge mode candidates from spatial and temporal neighboring blocks is shown.

[0022] Figure 3AAn example of an affine motion field of a block described by motion information of two control points (4 parameters) is shown.

[0023] Figure 3B An example of an affine motion field of a block described by motion information of three control point motion vectors (6 parameters) is shown.

[0024] Figure 4 An example of block-based affine transform prediction is shown, where the motion vector for each 4×4 luma sub-block is derived from the control point MV.

[0025] Figure 5 An example of spatially neighboring sub-blocks of the current CU used for RMVF (regression-based motion vector field) motion parameter derivation is shown.

[0026] Fig. 6A An example of deriving the CPMV of the current CU based on RVMF is shown, where the motion vector at the center of the current block is derived using the sub-block motion vector fields of non-neighboring CUs according to a regression model.

[0027] Figure 6B An example according to an embodiment of the present invention is shown, in which only three sub-block MVs (SBMVs) in non-neighboring CUs are used to derive the motion vector of the current CU.

[0028] Figure 7 An example according to an embodiment of the present invention is shown, in which the CPMV and spatially neighboring SBMV of non-adjacent CUs are used for regression model derivation.

[0029] Figure 8 Shows something like Figure 7 An example in which the SBMVs of non-adjacent CUs and spatially neighboring SBMVs are used for regression model derivation.

[0030] Fig. 9 An example of regression MV derivation according to an embodiment of the present invention is shown, where one regression-based MV is derived from a neighboring sub-block and another regression-based MV is derived from a non-neighboring CU, and the two derived MVs are mixed.

[0031] Fig.10 An example of regressive MV derivation according to an embodiment of the present invention is shown, in which CPMVs are derived from non-neighboring CUs and the CPMVs are mixed with neighboring SBMVs, respectively.

[0032] Fig.11 A flowchart of an exemplary video encoding and decoding system derived by using a simplified regression model according to an embodiment of the present invention is shown.

[0033] Fig.12A flowchart of another exemplary video encoding and decoding system using a simplified regression model inference according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0034] It will be readily appreciated that the components of the present invention as generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the embodiments of the systems and methods of the present invention as shown in the figures is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the present invention. References throughout this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in conjunction with that embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily all refer to the same embodiment.

[0035] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. However, those skilled in the relevant art will recognize that the present invention may be practiced without one or more specific details, or practiced using other methods, components, etc. In other cases, well-known structures or detailed descriptions of operations are not shown or are not shown to avoid obscuring aspects of the present invention. The illustrated embodiments of the present invention will be best understood with reference to the accompanying drawings, in which like parts are represented by like digits throughout. The following description is intended to be exemplary only, and simply illustrates certain selected embodiments of the apparatus and methods consistent with the present invention as claimed herein.

[0036] Affine Motion Compensated Prediction

[0037] In HEVC, only the translation motion model is used for motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensation prediction is applied. As shown in FIG. 3A-B, the affine motion field of block 310 is composed of Figure 3A The motion information of two control points (4 parameters) or Figure 3B The three control point motion vectors (6 parameters) are described in .

[0038] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0039]

[0040] For the 6-parameter affine motion model, the motion vector at a sample position (x, y) in the block is derived as:

[0041]

[0042] Where (mv 0x ,mv 0y ) is the motion vector of the upper left control point, (mv 1x ,mv 1y ) is the motion vector of the upper right control point, and (mv 2x ,mv 2y ) is the motion vector of the lower left control point.

[0043] To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luminance sub-block, the motion vector of the center sample of each sub-block is calculated according to the above equation, as Figure 4 As shown, and rounded to 1 / 16 fractional precision. A motion compensated interpolation filter is then applied to generate a prediction for each subblock with a derived motion vector. The subblock size of the chrominance component is also set to 4×4. The MV of a 4×4 chrominance subblock is calculated as the average of the MVs of the top left and bottom right luminance subblocks in the juxtaposed 8x8 luminance region.

[0044] As done for translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP mode.

[0045] Affine merge prediction

[0046] AF_MERGE (ie, affine merge) mode can be applied to CUs with width and height greater than or equal to 8. In this mode, the CPMV (control point MV) of the current CU is generated based on the motion information of spatially neighboring CUs. There can be up to five CPMVP (CPMV prediction) candidates, and the signal index is used to indicate which one to use for the current CU. The following three CPVM candidates are used to form the affine merge candidate list:

[0047] – Inherited affine merge candidates inferred from the CPMV of neighboring CUs

[0048] – Affine merged candidate CPMVP constructed using the translation MV of the neighboring CU

[0049] – Zero MV

[0050] A CU level flag is used to signal the affine merge mode as one sub-block merge mode, while the other sub-block merge mode indicates the SBTMvp mode.

[0051] Regression based Motion Vector Field (RMVF)

[0052] The motion behavior can vary within a block. In particular, for larger CUs, it is not efficient to represent the motion behavior with only one motion vector. The RMVF method models this motion behavior based on the motion vectors of spatially neighboring sub-blocks. The spatially neighboring sub-blocks of the current CU used for RMVF motion parameter derivation are as follows: Figure 5 The motion vectors and center positions of the neighboring sub-blocks of the current CU are used as inputs to the linear regression process to derive a set of linear model parameters by minimizing the mean square error, namely {a xx ,a xy ,a yx ,a yy ,b x ,b y}.exist Figure 5 , the current CU 510 is shown as a thick line rectangle, and the current CU is divided into sub-blocks shown as small empty squares. The spatial neighboring sub-blocks of the current CU are shown as dot-filled squares.

[0053] Then, with (X subPU ,Y subPU ) is the motion vector (MV X_subPU ,MV Y_subPU ) is calculated as:

[0054]

[0055] Finally, the derived regression model is used to refine the motion field of the current CU.

[0056] Using SBMV for candidate derivation of non-adjacent CUs

[0057] The concept of using the MV of non-adjacent CUs for translational merging of MV candidates and AMVP MV candidate derivation can also be applied to affine mode. The CPMV of non-adjacent affine coded CUs can be used to derive the CPMV of the current block for merging MV candidates and AMVPMV candidates. The following equation can be used to derive the CPMV of the current block:

[0058]

[0059] or

[0060]

[0061] In the above equation, (x, y) represents a specific point in the current block, which can be the position of the control point of the current block. 0x ,v 0y ) is the control point motion vector V at the upper left corner of the reference block 0 ,(v 1x ,v 1y ) is the control point motion vector V at the upper right corner of the reference block 1 ,(v 2x ,v 2y ) is the control point motion vector V at the lower left corner of the reference block 2 . (w,h) are the width and height of the reference block.

[0062] However, the stored CPMVs of non-adjacent CUs require a large amount of memory space, which is not friendly to hardware-based implementation. Therefore, it is proposed to use the sub-block MVs (SBMVs, MxN sub-block MVs, where M and N are positive integers, such as 4) of non-adjacent CUs to derive the CPMV of the current block instead of using the CPMVs of non-adjacent CUs. In addition, for deriving the CPMV of the current CU using the SBMVs of non-adjacent CUs, two or more SBMVs of the four corner subblocks of the non-adjacent CU or two or more SBMVs inside the non-adjacent CU are used to derive the CPMV of the current block. When the SBMVs of the corner subblocks of the reference block are used for derivation, w and h can still be equal to the width and height of the reference block, or equal to (width-M) and (height-N), or equal to (width>>1) and (height>>1) of the reference block.

[0063] In one embodiment, for affine CPMV derivation, candidates from adjacent CUs may be derived using CPMV, and candidates from non-adjacent CUs may be derived using SBMV. In another embodiment, for affine CPMV derivation, candidates from adjacent CUs or within a derived or predefined area may be derived using CPMV; otherwise, candidates from non-adjacent CUs and outside the area may be derived using SBMV. The area may be the current CTU, or the current CTU and the left CTU, or a PxQ area (P and Q are positive integers), or multiple virtual processing data units (VPDUs). In one example, for adjacent CUs in the upper CTU row, the left CTU column, or CTUs in different slices / tiles, only selected SBMVs (e.g., corner SBMVs) are used.

[0064] Fig. 6A An example of deriving the CPMV of the current CU based on the RVMF is shown, in which the motion vector at the center of the current block 610 is derived using the sub-block motion vector field of the non-neighboring CU 620 according to the regression model. Figure 6B An example according to an embodiment of the present invention is shown, in which only three subblock MVs (SBMVs) in a non-adjacent CU 620 are used instead of all subblock MVs to derive the motion vector of the current CU 610. Since the subblock MVs are already stored in the codec system, the use of the subblock MVs does not require any additional storage. Using the three SBMVs of the non-adjacent CUs to derive the MV of the current block can greatly reduce the derivation based on the regression model.

[0065] Regression-based affine candidate derivation

[0066] As mentioned above, the RMVF method takes the motion vectors of spatially neighboring sub-blocks as input to derive a regression model for current CU motion field refinement. The concept of non-adjacent affine candidate derivation can be combined with the RMVF method.

[0067] In one embodiment, the motion vectors of reconstructed spatially neighboring sub-blocks and the motion information of non-adjacent affine CUs (e.g., sub-block motion vectors or CPMVs) or the motion information of the current CU derived from the affine parameters of the non-adjacent affine CUs are used as input to derive a regression model. By using the regression model, the CPMV of the current block can be derived. For N non-adjacent affine CUs, we can derive N CPMV candidates for the current block. When the CPMV of a non-adjacent affine CU or the CPMV of the current block derived from a non-adjacent affine CU is used for regression, the weight of the CPMV should be increased because the CPMV can represent all sub-blocks of the non-adjacent CU or all sub-blocks of the current block. However, in this method, for N non-adjacent affine CUs, N regression processes need to be performed.

[0068] In order to use the spatially neighboring SBMVs and the motion information of non-neighboring CUs for regression, in one example, the SBMVs of non-neighboring CUs are used. Figure 7 In another example shown, the CPMV of non-adjacent CUs and spatially adjacent SBMVs are used. In one embodiment, for using the CPMV of non-adjacent CUs, the CPMV of non-adjacent CUs is used to derive the SBMV of the current block. The SBMV of the current block derived based on the CPMV 725 of the non-adjacent CU and the spatially adjacent SBMV 735 are used for regression, such as Figure 7 As shown, where the spatially neighboring sub-blocks 730 of the current block 710 are shown as dot-filled squares, and the block 720 corresponds to a non-neighboring CU encoded and decoded in affine mode. In another embodiment, the CPMV of the non-neighboring CU is used to derive the CPMV of the current block. The derived CPMV of the current block is given a higher weight and combined with the spatially neighboring SBMV for regression. In another embodiment, the CPMV of the non-neighboring CU is weighted with a higher weight and combined with the spatially neighboring SBMV for regression. Although Figure 7The case where all spatially neighboring SBMVs are used for regression is shown, but the present invention can also use some spatially neighboring SBMVs for regression.

[0069] For SBMV using non-adjacent CUs, Figure 8 Another embodiment shown uses the SBMVs of non-neighboring CUs and spatially neighboring SBMVs for regression. Figure 8 The method shown is similar to Figure 7 However, in Figure 8 , the SBMV 825 of the upper left, upper right and lower left sub-blocks, instead of Figure 7 The CPMV 725 of the non-adjacent CU 720 in is used for regression. In another embodiment, the corner SBMV of the non-adjacent CU is used as the CPMV of the non-adjacent CU to derive the SBMV of the current block. The derived SBMV and spatially neighboring SBMV of the current block are used for regression. In another embodiment, the corner SBMV of the non-adjacent CU is used as the CPMV of the non-adjacent CU to derive the CPMV of the current block. The derived CPMV of the current block is given a higher weight and combined with the spatially neighboring SBMV for regression. In another embodiment, two or more SBMVs of non-adjacent CUs are used to derive the CPMV of the non-adjacent CU, to derive the CPMV of the current block, or to derive the CPMV of the non-adjacent CU. The derived CPMV is given a higher weight and combined with the spatially neighboring SBMV for regression.

[0070] In another invention, we propose to use the motion vectors of spatially neighboring sub-blocks as input to derive a regression model (denoted as Mn). In addition, the motion information of non-adjacent affine CUs (e.g., sub-block motion vectors or CPMVs) or the motion information of the current CU derived from the affine parameters of the non-adjacent affine CUs is used to derive a set of regression models (denoted as {Ma1, Ma2,…, MaN}, where N = the number of non-adjacent affine CUs). By mixing Mn with {Ma1, Ma2,…, MaN} through a certain derivation method, a final set of regression models (denoted as {Mf1, Mf2,…, MfN}) can be obtained and used to refine the motion field in the CU. The final set of regression models {Mf1, Mf2,…, MfN} can be used to derive N CPMV candidates for the current block, such as Fig. 9 As shown. In this method, regression only needs to be performed once. Affine parameter mixing can use information related to the number of neighboring sub-blocks for spatial reconstruction, the number of sub-blocks of non-neighboring CUs, the number of sub-blocks of the current block, the distance between the non-neighboring CU and the current block, and any combination thereof.

[0071] like Fig. 9 As shown, a MV (expressed as ). Another MV (expressed as ) corresponds to the regression model derived from the sub-block MV of the non-adjacent CU 920. Fig. 9 For simplicity, only one non-adjacent affine-coded CU and only one derived MV are shown. However, there may be N non-adjacent affine-coded CUs and N derived MVs In addition, SBMV 932 from some neighboring sub-blocks instead of all neighboring sub-blocks may be used (in Fig. 9 In the example of FIG. 9 , only three SBMVs are used to derive the regression-based MV 935 of the neighboring area 930. Similarly, some sub-blocks ( Fig. 9 The inherited MV 925 of a CU (only two sub-blocks are shown in FIG. 1 ) may be used to derive the regression-based MV 925 of a non-adjacent CU.

[0072] In one embodiment, a regression model (denoted as Mn) is derived from the motion vectors of spatially neighboring sub-blocks, and the CPMV set of the current CU (denoted as Cn) can be derived from Mn. On the other hand, several CPMV sets (denoted as {Ca1, Ca2, ..., CaN}, where N = the number of non-adjacent affine CUs) can be obtained or derived from non-adjacent affine CUs. By weighted mixing Cn and {Ca1, Ca2, ..., CaN} according to block distance and / or block size or predefined weights of Cn and {Ca1, Ca2, ..., CaN}, multiple final CPMV sets (e.g., {Cf1, Cf2, ..., CfN}, where N = the number of non-adjacent affine CUs) can be obtained and used to refine the motion field in the CU.

[0073] In another embodiment, a CPMV set, an affine parameter set, or a regression model is derived from the motion vectors of spatially neighboring sub-blocks. One or more CPMV sets, affine parameter sets, or regression models are obtained or derived from non-neighboring affine CUs. Using the CPMV / affine parameter / regression model, the MV field of the current block derived from the spatially neighboring sub-blocks and the MV field of the current block derived from the non-neighboring CU can be derived. The two motion fields can be mixed at the sub-block MV level according to the position within the CU, such as Fig. 9 As shown. For example, a subblock close to the CU boundary may have a higher weight on the MV field derived from the spatially neighboring subblock, and a subblock (SB) far from the CU boundary may have a higher weight on the MV field derived from the non-adjacent CU. In addition, the opposite approach may also be applied (for example, a subblock close to the CU boundary may have a lower weight on the MV field derived from the spatially neighboring subblock, and a subblock far from the CU boundary may have a lower weight on the MV field derived from the non-adjacent CU).

[0074] In one invention, one or more CPMV sets, affine parameter sets or regression models are obtained or derived from a non-adjacent affine CU by SBMV, CPMV or regression. One or more novel affine candidates can be derived by mixing the CPMV set, affine parameter set or regression model with the upper left, upper right and lower left neighboring MVs. For example, the upper left CPMV derived from a non-adjacent affine CU can be mixed with one or more upper left neighboring MVs. The upper right CPMV derived from a non-adjacent affine CU can be mixed with one or more upper right neighboring MVs. Moreover, the lower left CPMV derived from a non-adjacent affine CU can be mixed with one or more lower left neighboring MVs. Each CPMV can be mixed with adjacent MVs respectively.

[0075] In one embodiment, one or more CPMV sets, affine parameter sets or regression models are obtained or derived from a non-adjacent affine CU by SBMV, CPMV or regression. One or more novel affine candidates can be derived by mixing the CPMV set, affine parameter set or regression model with the affine models derived from the upper left, upper right and lower left neighboring MVs. For example, the upper left CPMV derived from a non-adjacent affine CU can be mixed with the affine model derived from the upper left neighboring MV. The upper right CPMV derived from a non-adjacent affine CU can be mixed with the affine model derived from the upper right neighboring MV. In addition, the lower left CPMV derived from a non-adjacent affine CU can be mixed with the affine model derived from the lower left neighboring MV. Each CPMV can be mixed with the affine models derived from the neighboring MVs, respectively. Fig.10 An example of this embodiment is shown in , where CPMVs 1032, 1042, and 1052 of the current CU 1010 are derived based on the non-adjacent CU 1020. The derived CPMVs 1032, 1042, and 1052 are mixed with the lower left, upper left, and upper right and neighboring MVs of the neighboring regions 1030, 1040, and 1050, respectively. For example, the CPMV 1042 is mixed with the SBMV in the upper left region 1040.

[0076] Any method or process presented herein may be combined. In addition, for the SBMV of non-adjacent CUs, they may be subsampled. For example, the SBMV may be subsampled by 2 / 4 / 8 / 16 / 32 / 64 / N in the x and y directions, respectively, where N is a positive integer. In another example, the SBMV may be subsampled by 2x2 / 4x4 / 8x8 / 16x16 / 32x32 / 64x64 / NxN in the x and y directions, respectively, where N is a positive integer. In one embodiment, a subsampled SBMV is used for non-adjacent CUs other than the current CTU / CTU row, or the current CTU and the left CTU, or a derived / predefined area, or any combination of the above. Otherwise, a non-subsampled SBMV may be used.

[0077] Any motion vector derivation method for the above affine merge mode can be implemented in an encoder and / or decoder. For example, any of the proposed methods can be implemented in an inter-frame codec module of an encoder (e.g., Figure 1A inter-frame prediction 112 in the decoder), the motion compensation module of the decoder (e.g., Figure 1B 152), or implemented in a merge candidate derivation module in an encoder or decoder, where a CPMV, an affine parameter set, or a regression model can be derived and used to generate a mixed MV to be inserted into a merge candidate list. Alternatively, any of the proposed methods can be implemented as a circuit coupled to the inter-frame coding and decoding module of the encoder and / or the motion compensation module, merge candidate derivation module of the decoder. Although the inter-frame prediction 112 and MC 152 are shown as independent processing units that support the motion vector derivation method of the affine merge mode, they can correspond to executable software or firmware code stored on a medium (such as a hard disk or flash memory) for a CPU (central processing unit) or a programmable device (such as a DSP (digital signal processor) or an FPGA (field programmable gate array)).

[0078] Fig.11 A flowchart of an exemplary video codec system derived using a simplified regression model according to an embodiment of the present invention is shown. The steps shown in the flowchart can be implemented as executable program codes on one or more processors (e.g., one or more CPUs) on the encoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, pixel data associated with a current block on the encoder side or encoded data associated with a current block to be decoded on the decoder side is received in step 1110. In step 1120, one or more CPMVs (control point motion vectors), affine parameter sets, or regression models of the current block are derived based on non-adjacent affine codec blocks. In step 1130, at least one of the one or more CPMVs, at least a portion of the affine parameter set, or at least a portion of the regression model is mixed with a neighboring MV (motion vector) of a sub-block in a neighboring area to generate at least one mixed MV (motion vector), and the neighboring MV of the sub-block in the neighboring area is associated with the at least one of the one or more CPMVs, the at least a portion of the affine parameter set, or the at least a portion of the regression model. In step 1140, the CPMV set including the at least one mixed MV is inserted into the merge candidate list. In step 1150, the current block is encoded or decoded using information including the merge candidate list.

[0079] Fig.12A flowchart of another exemplary video codec system derived using a simplified regression model according to an embodiment of the present invention is shown. According to the method, in step 1210, pixel data associated with a current block at the encoder side or encoded data associated with a current block to be decoded at the decoder side is received. In step 1220, a subset of subblocks of non-adjacent affine codec blocks is determined, wherein the subset of subblocks is less than the total number of subblocks in the non-adjacent affine codec blocks. In step 1230, a set of CPMVs (control point motion vectors) or SBMVs (subblock motion vectors) associated with a subset of subblocks of the current block are derived based on the subset of subblocks of the non-adjacent affine codec blocks. In step 1240, one or more derived MVs (motion vectors) are derived for the current block based on regression processing using an input MV, wherein the input MV includes the set of CPMVs or SBMVs associated with a subset of subblocks of the non-adjacent affine codec blocks. In step 1250, one or more derived MVs are inserted into a merge candidate list. In step 1260, the current block is encoded or decoded using the information including the merge candidate list.

[0080] The flowchart shown is intended to illustrate an example of video coding according to the present invention. Without departing from the spirit of the present invention, each step can be modified, the steps can be rearranged, the steps can be split or combined to implement the present invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples to implement embodiments of the present invention. Without departing from the spirit of the present invention, those skilled in the art can implement the present invention by replacing syntax and semantics with equivalent syntax and semantics.

[0081] The above description is provided to enable those skilled in the art to practice the present invention provided in the context of a specific application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but to conform to the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details are illustrated to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention may be implemented.

[0082] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, one embodiment of the present invention can be one or more electronic circuits integrated into a video compression chip or a program code integrated into the video compression software to perform the processing described herein. The embodiments of the present invention can also be a program code to be executed on a digital signal processor (DSP) to perform the processing described herein. The present invention can also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software codes or firmware codes that define the specific methods embodied by the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. It is also possible to compile software code for different target platforms. However, the different code formats, styles and languages ​​of the software code and other ways of configuring the code to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0083] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered in all respects as illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes that fall within the meaning and range of equivalents of the claims should be included within their scope.

Claims

1. A video encoding and decoding method, the method comprising: Receiving pixel data associated with a current block at an encoder side or receiving encoded data to be decoded associated with the current block at a decoder side; Deriving one or more control point motion vectors, affine parameter sets or regression models of the current block based on non-adjacent affine codec blocks; blending at least one of the one or more control point motion vectors, at least a portion of the affine parameter set, or at least a portion of the regression model with a neighboring motion vector of a sub-block within a neighboring region to generate at least one blended motion vector, the neighboring motion vector of the sub-block within the neighboring region being associated with the at least one of the one or more control point motion vectors, the at least a portion of the affine parameter set, or the at least a portion of the regression model; inserting a control point motion vector set including the at least one blended motion vector into a merge candidate list; as well as The current block is encoded or decoded using information including the merge candidate list.

2. The method according to claim 1, characterized in that The one or more control point motion vectors correspond to a lower left control point motion vector, an upper left control point motion vector, and an upper right control point motion vector.

3. The method according to claim 2, characterized in that A lower-left mixed motion vector is generated for the control point motion vector, an upper-left mixed motion vector is generated for the upper-left control point motion vector, and an upper-right mixed motion vector is generated for the upper-right control point motion vector.

4. The method according to claim 1, characterized in that: The at least one blended motion vector is generated using a weighted sum of the at least one of the one or more control point motion vectors and the neighboring motion vector.

5. The method according to claim 4, characterized in that The weight of this weighted sum depends on the block distance, the block size, or both.

6. The method according to claim 4, characterized in that The weights of the weighted sum are predefined.

7. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors, arranged to: Receiving pixel data associated with a current block at an encoder side or receiving encoded data to be decoded associated with the current block at a decoder side; Deriving one or more control point motion vectors, affine parameter sets or regression models of the current block based on non-adjacent affine codec blocks; blending at least one of the one or more control point motion vectors, at least a portion of the affine parameter set, or at least a portion of the regression model with a neighboring motion vector of a sub-block within a neighboring region to generate at least one blended motion vector, the neighboring motion vector of the sub-block within the neighboring region being associated with the at least one of the one or more control point motion vectors, the at least a portion of the affine parameter set, or the at least a portion of the regression model; inserting a control point motion vector set including the at least one blended motion vector into a merge candidate list; as well as The current block is encoded or decoded using information including the merge candidate list.

8. A video encoding and decoding method, the method comprising: Receiving pixel data associated with a current block at an encoder side or receiving encoded data to be decoded associated with the current block at a decoder side; Determining a subset of subblocks of a non-adjacent affine codec block, wherein the subset of the subblocks is less than a total number of subblocks in the non-adjacent affine codec block; deriving a set of control point motion vectors or sub-block motion vectors associated with the subset of sub-blocks of the current block based on the subset of sub-blocks of the non-adjacent affine codec block; determining one or more derived motion vectors for the current block based on a regression process using input motion vectors, wherein the input motion vectors include the set of control point motion vectors or sub-block motion vectors associated with the subset of sub-blocks of the non-adjacent affine codec block; inserting the one or more derived motion vectors into a merge candidate list; as well as The current block is encoded or decoded using information including the merge candidate list.

9. The method according to claim 8, characterized in that The set of control point motion vectors or sub-block motion vectors corresponds to a lower left sub-block motion vector, an upper left sub-block motion vector, and an upper right sub-block motion vector.

10. The method according to claim 9, characterized in that A derived motion vector of the current block is determined based on the lower-left sub-block motion vector, the upper-left sub-block motion vector, and the upper-right sub-block motion vector.

11. The method according to claim 8, characterized in that The input motion vector also includes sub-block motion vectors of neighboring sub-blocks of the current block.

12. The method according to claim 11, characterized in that A first derived motion vector of the current block is determined based on the set of control point motion vectors or sub-block motion vectors associated with the subset of the sub-blocks of the non-adjacent affine codec block, a second derived motion vector of the current block is determined based on sub-block motion vectors of neighboring sub-blocks of the current block, and a final derived motion vector is generated by blending the first derived motion vector and the second derived motion vector.

13. The method according to claim 12, characterized in that The final derived motion vector corresponds to a weighted sum of the first derived motion vector and the second derived motion vector, and the first derived motion vector has a higher weight than the second derived motion vector.

14. An apparatus for video encoding and decoding, the apparatus comprising one or more electronic devices or processors, arranged to: Receiving pixel data associated with a current block at an encoder side or receiving encoded data to be decoded associated with the current block at a decoder side; Determining a subset of subblocks of a non-adjacent affine codec block, wherein the subset of the subblocks is less than a total number of subblocks in the non-adjacent affine codec block; deriving a set of control point motion vectors or sub-block motion vectors associated with the subset of sub-blocks of the current block based on the subset of sub-blocks of the non-adjacent affine codec block; determining one or more derived motion vectors for the current block based on a regression process using input motion vectors, wherein the input motion vectors include the set of control point motion vectors or sub-block motion vectors associated with the subset of sub-blocks of the non-adjacent affine codec block; inserting the one or more derived motion vectors into a merge candidate list; as well as The current block is encoded or decoded using information including the merge candidate list.