A low-complexity affine motion estimation method based on AME

Abstract

The present invention belongs to the field of video coding, and discloses a low-complexity affine motion estimation method for AME, which includes the following steps: Step 1: Early prediction of the optimal inter-frame mode based on the partition depth: Predict the optimal inter-frame mode of the child CU according to the optimal inter-frame mode of the parent CU; Step 2: Low-complexity algorithm inside AME: Inside the AME algorithm, accelerate the AME algorithm by whether CPMV is parallel, early termination of the iteration process, and whether it is necessary to perform fine-grained adjustment and optimization; Step 3: Low-complexity algorithm outside AME: Extract relevant features for the CU that needs to make a mode decision between traditional Inter and AME, and based on the decision tree, make an early prediction of its optimal inter-frame mode for such a CU, so as to skip the RDO decision and the traversal process of sub-optimal modes. The present invention greatly reduces the complexity of the VVC inter-frame mode decision.

Description

Technical Field

[0001] The invention belongs to the field of video coding, and in particular relates to an AME low-complexity affine motion estimation method. Background Art

[0002] As the demand for 8K, 16K and other ultra-high-definition videos increases, the field of video coding is developing rapidly. In 2020, the Joint Video Exploration Team (JVET) finalized the latest generation of Versatile Video Coding (VVC), which is nearly 50% more efficient than the previous generation High Efficiency Video Coding (HEVC), which helps to cope with the thorny problem of high-resolution video storage.

[0003] While compression efficiency has been improved, the cost is a sharp increase in encoder complexity. The main reason is the introduction of a number of new technologies, including: increased intra-frame prediction angles, increased inter-frame modes, secondary transforms, multi-transform kernel selection, dependency quantization and other new technologies. The complexity of the inter-frame prediction part is extremely high. In addition to selecting the optimal reference frame through rate distortion optimization (RDO), several inter-frame modes must be traversed and selected based on RDO within the inter-frame prediction.

[0004] Considering the complexity of inter-frame prediction, the present invention starts from the inter-frame prediction part. Since many video scenes have non-rigid motions such as rotation and scaling, traditional integer-pixel motion estimation (IME) and fractional-pixel motion estimation (FME) cannot describe these motions well. In response to this, VVC introduced a new affine motion estimation (AME) to specifically describe scenes such as rotation and scaling. However, AME itself is extremely complex, and AME compares with traditional IME and FME based on RDO to select the optimal mode, which makes the complexity of inter-frame prediction rise sharply. Summary of the invention

[0005] The present invention aims to provide an AME low-complexity affine motion estimation method to solve the above-mentioned technical problems.

[0006] In order to solve the above technical problems, a specific technical solution of an AME low-complexity affine motion estimation method of the present invention is as follows:

[0007] An AME low-complexity affine motion estimation method comprises the following steps:

[0008] Step 1: Advance prediction of the optimal inter-frame mode based on the partition depth: predict the optimal inter-frame mode of the sub-CU according to the optimal inter-frame mode of the parent CU;

[0009] Step 2: Low-complexity algorithm within AME: Within the AME algorithm, the AME algorithm is accelerated by checking whether the CPMV is parallel, the early termination of the iteration process, and whether fine-grained adjustment optimization is necessary;

[0010] Step 3: Low-complexity algorithm outside AME: For CUs that need to make mode decisions between traditional Inter and AME, relevant features are extracted, and the optimal inter-frame mode of such CUs is predicted in advance based on the decision tree, thereby skipping the RDO decision and the traversal process of the suboptimal mode.

[0011] Furthermore, the step 1 includes the following specific steps:

[0012] For AI-CU, if the optimal inter-frame mode of the current division depth is the traditional Inter, the next division depth, that is, the optimal inter-frame mode of the sub-CU is directly selected as the traditional Inter, skipping the traversal of AME and the comparison and decision-making process.

[0013] Furthermore, the step 2 includes the following specific steps:

[0014] Step 2.1: Early termination based on CPMV: In the AME process, CPMV is used to describe the rotated and scaled blocks, and then the motion vector of each small prediction block in the current CU is obtained based on CPMV, and finally motion compensation is performed to obtain the prediction block;

[0015] Step 2.2: Early termination of the iteration process: If the two CPMVs of the four parameters are parallel before the AME iteration process begins, the traditional Inter is directly selected as the optimal inter-frame mode, otherwise, the gradient-based iteration process of AME is performed;

[0016] Step 2.3: Fine-grained optimization: VTM10.0 adds 8 directions around the CU to determine whether the MV should be adjusted based on RDO, and obtains the final optimal CPMV.

[0017] Furthermore, the step 2.1 includes the following specific steps:

[0018] For AI-CU, if the two CPMVs of the four-parameter AME are parallel before AME starts to iterate, the AME process is skipped directly and the traditional Inter is used as the optimal inter-frame mode.

[0019] Furthermore, the step 2.2 includes the following specific steps:

[0020] During the iteration process within AME, the number of iterations is related to the frame type, four-parameter AME and six-parameter AME. The specific number of iterations is calculated as follows:

[0021]

[0022] For four-parameter and six-parameter AME, three iterations are required under the Low Delay B configuration to obtain the CPMV that minimizes the mean square error between the current CU and the reference CU; during the iteration, the sobel operator is used to calculate the horizontal and vertical gradients of the predicted CU, and then dMv is calculated based on the error matrix between the current block and the predicted block to obtain the CPMV of the next iteration, that is,

[0023]

[0024] CPMV i+1 =CPMV i +dMv i ,i∈[1,iterTime-1]

[0025] When the CPMV is predicted to be optimal, the iteration process is directly skipped;

[0026] The mean value of the error matrix piError, the width, height and absolute error sum of the current CU are used to determine early termination;

[0027] The distortion and error matrices of CUs of different sizes are quite different, so they are unified, that is,

[0028] T=D / (W*H*MSE)

[0029] Where D represents the SAD obtained by compensating with the current CPMV, MSE represents the mean of the error matrix of the current iteration process, W and H represent the width and height of the current CU respectively, and T represents the unified value.

[0030] If in a certain iteration process, T in the current iteration process is less than T in the previous iteration process, the iteration is stopped directly, and the information of the previous iteration is saved as the optimal information in the entire iteration process, that is,

[0031]

[0032] Furthermore, the step 2.3 includes the following specific steps:

[0033] Based on the preliminary optimization of VTM10.0, further optimization is performed by inserting two 1 / 2 pixels at the lower left corner and upper right corner of the current pixel, and using these two sub-pixels to replace the search process of 4 whole pixels above, below, left and right. If the RDO cost of these two sub-pixels after motion compensation is less than the RDO cost after the iteration is completed, it is considered necessary to perform fine-grained adjustment, otherwise the fine-grained adjustment process is skipped directly.

[0034] Furthermore, the step 3 includes the following specific steps:

[0035] Step 3.1: Use the decision tree model for offline training and define the true positive rate TPR, false positive rate FPR, precision rate PPV, false positive rate FDR, and false negative rate FNR. These parameters are defined as follows:

[0036]

[0037] Where TP represents the optimal inter-frame mode is affine, the number of CU blocks detected as affine, FN represents the number of affine predicted as inter, FP represents the number of inter predicted as affine, and TN represents the number of inter predicted as inter;

[0038] Step 3.2: Find relevant features based on the texture of the rotated and scaled CU, the relationship between the optimal inter-frame mode of the parent and child CU in the block partition, and the transformed energy theory;

[0039] Step 3.3: Use the standard deviation to remove the background CU with a small probability of selecting AME as the optimal inter-frame mode. For this, extract the original CU pixels and calculate their standard deviation.

[0040] Furthermore, the step 3.3 includes the following specific steps:

[0041] Step 3.3.1: Based on the standard deviation, the gradient is introduced to describe the texture complexity characteristics in more detail; for this, the horizontal gradient, the 45-degree gradient, the ratio of the horizontal to the vertical gradient, and the ratio of the 45-degree to the 135-degree gradient are added;

[0042] Step 3.3.2: The absolute transformation error and is used as a main variable to describe the transformation, reflecting the energy of the transformed CU, that is,

[0043]

[0044] Where: Represents the energy of the transformed CU, m and n represent the width and height of the CU respectively, c(i,j) represents the transform coefficient at position (i,j), and normalizes part of the energy of the last divided depth CU;

[0045] The partial energy of different TU SIZE is expressed as follows,

[0046]

[0047] In the formula, c(P k ) indicates that in P k The transformation coefficient of the position, n is a variable related to the TU size, that is, n = log2M, M is the TU size, and the CU partial energy is normalized, that is,

[0048]

[0049] The AME low-complexity affine motion estimation method of the present invention has the following advantages: the present invention optimizes the AME algorithm in all aspects: analyzes the inter-frame mode relationship between the parent and child CUs; ​​optimizes and accelerates the AME algorithm from three perspectives; and predicts the optimal inter-frame mode of the CU based on the decision tree model outside the AME algorithm. The complexity of VVC inter-frame mode decision is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a low-complexity AME flow chart of the present invention;

[0051] Figure 2 It is a relationship diagram of the optimal inter-frame mode of the father-child CU of the present invention;

[0052] Figure 3 Schematic diagram of CPMV for two inter-frame modes;

[0053] Figure 4 It is a diagram showing the parameter changes of the four-parameter AME in a certain iteration;

[0054] Figure 5 It is a schematic diagram of fine-grained search optimization of the present invention;

[0055] Figure 6 This is a schematic diagram of the improved fine-grained search optimization of the present invention;

[0056] Figure 7 A schematic diagram showing the cost comparison of fine-grained search optimization before and after the improvement of the present invention;

[0057] Figure 8 It is a schematic diagram of the decision tree structure of the present invention;

[0058] Fig. 9 It is a schematic diagram of the decision tree ROC curve of the present invention;

[0059] Fig.10 It is a schematic diagram of the decision tree confusion matrix of the present invention;

[0060] Fig.11 It is a schematic diagram for comparing the standard deviations of the two inter-frame modes;

[0061] Fig.12 Schematic diagram of gradient comparison between two inter-frame modes;

[0062] Fig.13 is a position distribution diagram of the transformation coefficients of the present invention;

[0063] Fig.14 Schematic diagram of partial energy normalization comparison of two inter-frame modes under different QPs. DETAILED DESCRIPTION

[0064] In order to better understand the purpose, structure and function of the present invention, the following further describes in detail an AME low-complexity affine motion estimation method of the present invention in conjunction with the accompanying drawings.

[0065] In order to more conveniently present the CU targeted by the present invention, the following definitions of CU are given before this:

[0066] Affine CU (A-CU): CU that directly performs AME in inter-frame mode, and uses AME as the optimal inter-frame mode;

[0067] Inter CU (I-CU): directly performs CU of traditional IME and FME in inter-frame mode, and takes traditional IME and FME as the optimal inter-frame mode;

[0068] Affine and Inter CU (AI-CU): AME, traditional IME, and FME are traversed in the inter-frame mode, and finally the optimal inter-frame mode is selected from the two based on RDO.

[0069] Since A-CU and I-CU do not have the process of RDO comparison with other inter-frame modes, AI-CU can be subdivided into the following two categories:

[0070] AI-A-CU: AME, traditional IME, and FME are all traversed in inter-frame mode, and finally AME is selected as the optimal inter-frame mode based on RDO;

[0071] AI-I-CU: AME and traditional IME and FME are traversed in inter-frame mode, and finally traditional IME and FME are selected as the optimal inter-frame mode based on RDO.

[0072] The AME algorithm acceleration and inter-frame mode decision optimization of the present invention are performed for A-CU and AI-CU respectively. The overall algorithm flow is as follows: Figure 1 shown.

[0073] 1. Early prediction of optimal inter-frame mode based on partition depth

[0074] In the process of block partitioning, each time a CU is partitioned, the RDO cost of the current partition is compared with the RDO cost of the previous depth partition to decide whether to perform the current partition. The selection of the optimal inter prediction mode is an indispensable part of the block partition decision and has an important impact on whether to perform the current partition decision. In this regard, the impact of the optimal inter prediction mode of the current partition depth on the optimal inter prediction mode of the next partition depth is considered, that is, the relationship between the child CU and the parent CU.

[0075] like Figure 2 As shown in the figure, the relationship between the inter-frame modes of the sub-CU and the parent CU of the video sequences "Cactus" and "RaceHorses" is given. It can be found that when the optimal inter-frame mode of the parent CU is Inter, the probability that the sub-CU selects the traditional Inter as the optimal inter-frame mode is extremely high. Therefore, for AI-CU, if the optimal inter-frame mode of the current division depth is the traditional Inter, the next division depth, that is, the optimal inter-frame mode of the sub-CU is directly selected as the traditional Inter, skipping the traversal of AME and the decision-making process of comparing with it.

[0076] 2. Low complexity algorithm within AME

[0077] 2.1 Early Termination Based on CPMV

[0078] In the AME process, CPMV is used to describe the rotated and scaled blocks, and then the motion vector of each small prediction block in the current CU is obtained based on CPMV, and finally motion compensation is performed to obtain the prediction block. Figure 3 As shown in the figure, for rigid motion, that is, blocks that tend to select traditional Inter as the optimal inter-frame mode, if four-parameter AME is performed, it will be found that the CPMVs of the upper left corner and the upper right corner are parallel; while for non-rigid motion, that is, blocks that tend to select AME as the optimal inter-frame mode, the two CPMVs of four-parameter AME are not parallel. To verify the effectiveness, we counted the proportion of traditional Inter and Affine in the "RaceHorses", "Cactus" and "MarketPlace" video sequences under the condition of QP=32, if the CPMVs are parallel, as shown in Table 1.

[0079] Table 1 Optimal inter-frame mode based on CPMV parallelism

[0080]

[0081] It can be seen from Table 1 that in the case of CPMV parallelism, the probability that the optimal inter-frame mode of the CU at this partition depth is the traditional Inter is extremely high.

[0082] Therefore, for AI-CU, if the two CPMVs of the four-parameter AME are parallel before AME starts to iterate, the AME process is skipped directly and the traditional Inter is used as the optimal inter-frame mode.

[0083] 2.2 Early Termination of Iteration Process

[0084] If the two CPMVs of the four parameters are parallel before the AME iteration process begins, the traditional Inter is directly selected as the optimal inter-frame mode, otherwise, the gradient-based iteration process of AME is performed. The CPMV at the beginning of the iteration directly inherits the optimal MVP obtained based on the spatial domain. Each time an iteration is performed, the MV of the sub-block is calculated based on the current CPMV, and then the prediction block is obtained through motion compensation, and finally the RDO cost of the current iteration is obtained. In other words, each iteration is accompanied by the calculation and comparison process of motion compensation and RDO cost, which is one of the reasons for the high complexity of AME.

[0085] During the iteration process within AME, the number of iterations is related to the frame type, four-parameter AME and six-parameter AME. The specific number of iterations is calculated as follows:

[0086]

[0087] For four-parameter and six-parameter AME, three iterations are required to obtain the CPMV that minimizes the mean squared error (MSE) between the current CU and the reference CU under the Low Delay B configuration. During the iteration, the sobel operator is used to calculate the horizontal and vertical gradients of the predicted CU, and then dMv is calculated based on the error matrix between the current block and the predicted block to obtain the CPMV of the next iteration, that is,

[0088]

[0089] CPMV i+1 =CPMV i +dMv i ,i∈[1,iterTime-1]

[0090] However, VTM10.0 will not terminate early when the optimal CPMV is reached, but will only continuously update the optimal information during the iteration until the iteration ends. Therefore, some CUs have already reached the optimal CPMV in the middle of the iteration, that is, such CUs do not need to iterate according to the specified number of iterations.

[0091] To preliminarily solve this problem, VTM10.0 itself has a simple solution to this redundant calculation: if in a certain iteration process, the two dMvs of four parameters or the three dMvs of six parameters are all 0, it is considered unnecessary to continue the iteration and the iteration process is exited in advance.

[0092] However, the early termination of VTM10.0 is simple and rough, and cannot avoid the situation where dMv is non-zero. In this regard, the present invention adopts an advanced early termination scheme, which directly jumps out of the iterative process when it is predicted that CPMV reaches the optimum.

[0093] Essentially, the CPMV is calculated based on the error matrix piError, and then the rate distortion of the current CPMV is calculated through motion compensation. That is, whether the current CPMV is selected as the optimal CPMV is determined by comparing the RDO cost. Larger blocks often have larger distortions. For this reason, the mean value of the error matrix piError, the width and height of the current CU, and the Sum of Absolute Difference (SAD) are taken into account to make an early termination judgment.

[0094] Considering that the distortion and error matrices of CUs of different sizes are quite different, they are unified, that is,

[0095] T=D / (W*H*MSE)

[0096] Where D represents the SAD obtained by compensating with the current CPMV, MSE represents the mean of the error matrix of the current iteration process, W and H represent the width and height of the current CU respectively, and T represents the unified value.

[0097] like Figure 4 The figure shows the changes of T and MSE, as well as the changes of CPMV in the corresponding iteration process. Figure 4 The CPMV in is expressed as the mean of the absolute values ​​of the horizontal and vertical coordinate differences. Figure 4 From the changes of the parameters in the iteration process, we can find that when MSE is monotonically decreasing and the change of CPMV, i.e. dMv, is also monotonically decreasing during the iteration process, T is monotonically increasing. Fundamentally speaking, whether the current iteration is the optimal iteration is determined by comparing the RDO cost. Therefore, only MSE cannot fully reflect whether the current iteration is the optimal iteration. This requires us to consider more variables, and T covers distortion and MSE. In other words, it is reasonable to conclude that it is necessary to continue the iteration process by observing the changes in T.

[0098] Therefore, if in a certain iteration process, T in the current iteration process is less than T in the previous iteration process, the iteration is stopped directly, and the information of the previous iteration is saved as the optimal information in the entire iteration process, that is,

[0099]

[0100] 2.3 Necessity of Fine-Grained Optimization

[0101] After the gradient-based iteration process is completed, the optimal CPMV with the minimum RDO cost and the best prediction effect can be obtained. However, the CPMV after iteration may not be the MV that can finally represent the motion of the CU. In other words, this MV may need fine-grained adjustment. Therefore, VTM10.0 adds 8 directions around the CU to determine whether the MV is adjusted based on RDO, and obtains the final optimal CPMV, such as Figure 5 shown.

[0102] In the calculation process of determining whether to adjust the CPMV, it is possible that the RDO cost after fine-grained adjustment will be greater than the cost before adjustment, that is, for such a CU, fine-grained adjustment is unnecessary. In order to reduce unnecessary redundant calculations, VTM10.0 itself optimizes fine-grained adjustment, that is, firstly perform a preliminary search in the four directions of up, down, left and right. If there is a direction in these four directions that makes the final RDO cost less than the RDO cost completed in the previous iteration, the fine-grained adjustment is considered to be effective, and further search is continued in the four directions of lower left, upper left, upper right, and lower right. Otherwise, stop the fine-grained adjustment optimization and directly skip the search in the four directions of lower left, upper left, upper right, and lower right.

[0103] Even though the fine-grained search has been initially optimized, it is still necessary to perform motion compensation based on CPMV in the four directions and calculate the RDO cost, so the complexity still exists. In this regard, the present invention further optimizes the VTM10.0 based on the initial optimization. Figure 6 As shown in the figure, two 1 / 2 pixels are inserted at the lower left corner and the upper right corner of the current pixel, and these two sub-pixels are used to replace the search process of 4 whole pixels above, below, left and right. If the RDO cost of these two sub-pixels after motion compensation is less than the RDO cost after the iteration is completed, it is considered necessary to perform fine-grained adjustment, otherwise the fine-grained adjustment process is skipped directly.

[0104] like Figure 7 The figure shows a comparison of the RDO cost calculated based on the original search direction and the newly given sub-pixel search direction. It can be seen that the sub-pixel search and the original whole pixel search have the same effect on judging the necessity of fine-grained search, and can accurately judge whether to conduct further fine-grained search.

[0105] Therefore, the method of the present invention using two sub-pixels to replace four whole pixels to determine the necessity of fine-grained optimization is completely feasible, and the AME process is accelerated based on this.

[0106] 3. Low-complexity algorithms outside AME

[0107] In the inter-frame prediction process of VVC, the traditional IME, FME and AME are mainly used to make inter-frame mode decisions based on RDO. However, some CUs do not make inter-frame mode decisions, but directly perform traditional IME, FME or AME, namely the I-CU and A-CU mentioned above. On the contrary, some CUs will traverse both inter-frame modes and finally select the optimal inter-frame mode based on RDO, namely AI-CU. For AI-CU, there are additional redundant calculations, namely the calculation process of the suboptimal mode at the cost of RDO. In this regard, for AI-CU, a scheme is proposed to predict the optimal inter-frame mode in advance, so as to eliminate the traversal and comparison process of the suboptimal mode. Specifically, some features of AI-A-CU and AI-I-CU are found, and offline training is performed based on the decision tree model to directly predict the optimal inter-frame mode of AI-CU.

[0108] 3.1 Model Structure

[0109] The present invention adopts a decision tree model for offline training. The specific structure of the decision tree model is as follows: Figure 8 As shown, the corresponding ROC curve and confusion matrix are as follows Fig. 9 , Fig.10 To characterize the accuracy of the model, we define the True Positive Rate (TPR), False Positive Rate (FPR), Precision Rate (PPV), False Discovery Rate (FDR), and False Negative Rate (FNR). These parameters are defined as follows:

[0110]

[0111] Where TP represents the optimal inter-frame mode is affine, the number of CU blocks detected as affine, FN represents the number of CU blocks predicted as inter, FP represents the number of CU blocks predicted as affine, and TN represents the number of CU blocks predicted as inter. Fig. 9 , Fig.10 It can also be seen that our decision tree model achieves good classification results.

[0112] 3.2 Related features

[0113] In order to find the difference between AME and traditional Inter in order to find the optimal inter-frame mode of CU, the relevant features are found based on the theory of texture of rotated and scaled CU, the relationship between the optimal inter-frame modes of parent and child CU in block partitioning, and the transformed energy.

[0114] 3.2.1 Standard Deviation

[0115] In the inter-frame mode decision process, the CUs that are rotated and scaled are generally in motion blocks with complex textures. Some background blocks that do not contain motion scenes often choose the traditional Inter mode with a larger block size, and the texture of such background blocks is relatively simple. Therefore, the standard deviation can be used to eliminate background CUs with a low probability of selecting AME as the optimal inter-frame mode.

[0116] For this, extract the original CU pixels and calculate their standard deviation. Fig.11 As shown in FIG. 1 , the comparison of the standard deviation mean of AME and traditional Inter under 4 QPs is given. It can be found that the standard deviation can be used as a clue to roughly distinguish the CU of AME and traditional Inter.

[0117] 3.2.2 Gradient

[0118] On the basis of standard deviation, gradient is introduced to describe the texture complexity features in more detail. For this purpose, horizontal gradient, 45-degree gradient, horizontal-to-vertical gradient ratio, and 45-degree-to-135-degree gradient ratio are added. In order to verify the effectiveness of several gradients in distinguishing between traditional Inter and AME modes, the video sequence "RaceHorses" is used to count the results of 20 frames under Low Delay B and QP=32, as shown in the figure. Fig.12 shown.

[0119] It can be found that in 20 frames, 4 gradients can roughly distinguish the traditional Inter and AME modes. Therefore, these 4 gradients are used as another clue to predict the optimal Inter mode.

[0120] 3.2.3 Absolute transformation error and

[0121] In addition to looking for clues based on texture complexity, we also focus on the partition depth and transformation. In VVC, inter-frame prediction is performed along with block partitioning, that is, inter-frame prediction is performed on CUs of each partition depth and various partition sizes, and some information about the current partition depth is highly correlated with the information about the previous partition depth. At the same time, every time a partition decision is made, it must be transformed, and the transformation of the CU at the current partition depth can reflect the energy share of the low-frequency area of ​​the current CU, which is crucial for the mode decision of the next-level depth CU. Therefore, consider the transformation of the previous-level partition depth CU.

[0122] The Sum of Absolute Transformed Difference (SATD) is a main variable describing the transformation, which can reflect the energy of the transformed CU, that is,

[0123]

[0124] Where: Indicates the energy of the transformed CU, m and n represent the width and height of the CU respectively, and c(i,j) represents the transform coefficient at position (i,j). In order to more accurately reflect the energy in the low-frequency area, the energy of the last deep CU is normalized.

[0125] The partial energy of different TU SIZE is expressed as follows,

[0126]

[0127] In the formula, c(P k ) indicates that in P k The transform coefficients of the position, n is a variable related to the TU size, that is, n = log2M, M is the TU size. In addition, the coefficient position distribution in TU is as follows Fig.13 As shown. The CU part energy is normalized, that is,

[0128]

[0129] like Fig.14 The figure shows the comparison of the normalized results η of some energies of AI-A-CU and AI-I-CU. It can be seen that it is reasonable to predict the inter-frame mode of the current depth CU based on the SATD of the last CU division. SATD is often positively correlated with the size of the transform unit (TU), so the width and height of the parent CU are also considered as two other features.

[0130] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. An AME low-complexity affine motion estimation method, characterized in that: The steps include: Step 1: Advance prediction of the optimal inter-frame mode based on the partition depth: predict the optimal inter-frame mode of the sub-CU according to the optimal inter-frame mode of the parent CU; Step 2: Low-complexity algorithm within AME: Within the AME algorithm, the AME algorithm is accelerated by checking whether the CPMV is parallel, the early termination of the iteration process, and whether fine-grained adjustment optimization is necessary; Step 2.1: Early termination based on CPMV: In the AME process, CPMV is used to describe the rotated and scaled blocks, and then the motion vector of each small prediction block in the current CU is obtained based on CPMV, and finally motion compensation is performed to obtain the prediction block; Step 2.2: Early termination of the iteration process: If the two CPMVs of the four parameters are parallel before the AME iteration process begins, the traditional Inter is directly selected as the optimal inter-frame mode, otherwise, the gradient-based iteration process of AME is performed; Step 2.3: Fine-grained optimization: VTM10.0 adds 8 directions around the CU to determine whether the MV should be adjusted based on RDO, and obtains the final optimal CPMV; Step 3: Low-complexity algorithm outside AME: Extract relevant features for CUs that need to make mode decisions between traditional Inter and AME, and predict the optimal inter-frame mode for such CUs in advance based on the decision tree, thereby skipping the RDO decision and the traversal process of the suboptimal mode; Step 3.1: Use the decision tree model for offline training and define the true positive rate TPR, false positive rate FPR, precision rate PPV, false positive rate FDR, and false negative rate FNR. These parameters are defined as follows: Where TP represents the optimal inter-frame mode is affine, the number of CU blocks detected as affine, FN represents the number of affine predicted as inter, FP represents the number of inter predicted as affine, and TN represents the number of inter predicted as inter; Step 3.2: Find relevant features based on the texture of the rotated and scaled CU, the relationship between the optimal inter-frame mode of the parent and child CU in the block partition, and the transformed energy theory; Step 3.3: Use the standard deviation to remove the background CU with a small probability of selecting AME as the optimal inter-frame mode. For this, extract the original CU pixels and calculate their standard deviation.

2. The AME low-complexity affine motion estimation method according to claim 1, characterized in that: The step 1 comprises the following specific steps: For AI-CU, if the optimal inter-frame mode of the current division depth is the traditional Inter, the next division depth, that is, the optimal inter-frame mode of the sub-CU is directly selected as the traditional Inter, skipping the traversal of AME and the comparison and decision-making process.

3. The AME low-complexity affine motion estimation method according to claim 2, characterized in that: The step 2.1 includes the following specific steps: For AI-CU, if the two CPMVs of the four-parameter AME are parallel before AME starts to iterate, the AME process is skipped directly and the traditional Inter is used as the optimal inter-frame mode.

4. The AME low-complexity affine motion estimation method according to claim 2, characterized in that: The step 2.2 includes the following specific steps: During the iteration process within AME, the number of iterations is related to the frame type, four-parameter AME and six-parameter AME. The specific number of iterations is calculated as follows: For four-parameter and six-parameter AME, three iterations are required under the Low Delay B configuration to obtain the CPMV that minimizes the mean square error between the current CU and the reference CU; during the iteration, the sobel operator is used to calculate the horizontal and vertical gradients of the predicted CU, and then dMv is calculated based on the error matrix between the current block and the predicted block to obtain the CPMV of the next iteration, that is, CPMV i+1 =CPMV i +dMv i ,i∈[1,iterTime-1] When the CPMV is predicted to be optimal, the iteration process is directly skipped; The mean value of the error matrix piError, the width, height and absolute error sum of the current CU are used to determine early termination; The distortion and error matrices of CUs of different sizes are quite different, so they are unified, that is, T=D / (W*H*MSE) Where D represents the SAD obtained by compensating with the current CPMV, MSE represents the mean value of the error matrix of the current iteration process, W and H represent the width and height of the current CU respectively, and T represents the unified value; If in a certain iteration process, T in the current iteration process is less than T in the previous iteration process, the iteration is stopped directly, and the information of the previous iteration is saved as the optimal information in the entire iteration process, that is, 5. The AME low-complexity affine motion estimation method according to claim 1, characterized in that: The step 2.3 includes the following specific steps: Based on the preliminary optimization of VTM10.0, further optimization is performed by inserting two 1 / 2 pixels at the lower left corner and upper right corner of the current pixel, and using these two sub-pixels to replace the search process of 4 whole pixels above, below, left and right. If the RDO cost of these two sub-pixels after motion compensation is less than the RDO cost after the iteration is completed, it is considered necessary to perform fine-grained adjustment, otherwise the fine-grained adjustment process is skipped directly.

6. The AME low-complexity affine motion estimation method according to claim 1, characterized in that: The step 3.3 includes the following specific steps: Step 3.3.1: Based on the standard deviation, the gradient is introduced to describe the texture complexity characteristics in more detail; for this, the horizontal gradient, the 45-degree gradient, the ratio of the horizontal to the vertical gradient, and the ratio of the 45-degree to the 135-degree gradient are added; Step 3.3.2: The absolute transformation error and is used as a main variable to describe the transformation, reflecting the energy of the transformed CU, that is, Where: Represents the energy of the transformed CU, m and n represent the width and height of the CU respectively, c(i,j) represents the transform coefficient at position (i,j), and normalizes part of the energy of the last divided depth CU; The partial energy of different TU SIZE is expressed as follows, In the formula, c(P k ) indicates that in P k The transformation coefficient of the position, n is a variable related to the TU size, that is, n = log2M, M is the TU size, and the CU partial energy is normalized, that is,

Images