Fast affine motion estimation method based on reconstructed prior information

Abstract

The present invention starts from the mutual exclusivity between the VVC inter-frame affine motion estimation mode and the Skip mode, and the correlation between the best inter-frame prediction mode of the current CU and its parent CU, previous sub-CU and adjacent CU, and discloses a fast affine motion estimation method based on reconstructed prior information. First, the current best prediction mode of the current CU, the best prediction mode of the parent CU, previous sub-CU and adjacent CU are extracted, and unnecessary affine motion estimation processes are skipped in advance by determining whether the best mode is the Skip mode, thereby reducing the overall encoding time. Compared with the VVC coding standard, the VVC fast radial motion estimation method disclosed by the present invention significantly reduces the time complexity of VVC without significantly reducing the compression coding efficiency.

Description

Technical Field

[0001] The present invention relates to the technical problem of reducing the complexity of affine motion estimation in the field of video coding, and in particular to a fast affine motion estimation method based on reconstructed prior information. Background Art

[0002] With the development and widespread adoption of 5G technology, the use of ultra-high-definition video is becoming increasingly widespread, and video data is growing rapidly. However, transmission bandwidth and the storage capacity of various terminal devices are limited. Previous video compression coding technologies, H.264 / AVC and H.265 / HEVC, are no longer able to meet market demand. To research more efficient video compression coding technologies to support future generations of high-resolution video coding standards, the Joint Video Exploration Team (JVET), a joint effort of MPEG and ITU, officially released the next-generation video compression coding standard, H.266 / VVC, in July 2020. The standard reference software is VTM. VVC's overall coding framework is largely different from HEVC, but new technologies are introduced in each coding module. Without compromising visual video quality, VVC achieves approximately 50% higher coding efficiency than HEVC. However, this also significantly increases encoding time, hindering real-time communication.

[0003] In the inter-frame prediction module, VVC introduces an affine motion estimation model. Compared to HEVC's inter-frame prediction, which only uses translational motion to compensate for moving objects, the affine motion model can not only describe the translational motion of objects, but also their rotation and scaling, which is closer to the actual movement process of objects in reality. Affine motion estimation brings higher compression efficiency, but also brings high complexity, which greatly increases the prediction coding time and is not conducive to application in many practical scenarios. At the same time, VVC introduces multi-type tree partitioning in the coding block (CU) division part, and recursively selects the best inter-frame prediction mode, which further increases the amount of calculation and significantly increases the coding time. Therefore, it is meaningful to find a fast affine motion estimation method that barely affects VVC's compression efficiency.

[0004] In general, moving objects in a video only occupy a small portion of the entire frame, with the rest consisting mostly of the background. Typically, only the moving object's trajectory involves translation, rotation, and scaling, while the background generally only experiences translational motion. During inter-frame prediction in VVC, affine motion estimation is performed on all regions, but only a small fraction of them ultimately use the affine motion estimation mode as the optimal inter-frame prediction mode. This results in a significant amount of unnecessary computation, wasting time.

[0005] In the process of selecting the best inter-frame prediction mode for a CU, VVC recursively calculates the rate-distortion cost of the CU partition and mode. By calculating the rate-distortion cost of each mode under each partition method, the one with the lowest cost is selected as the best mode. The rate-distortion cost calculation formula is shown in formula (1):

[0006] J=SAD+λ×B (1)

[0007] Among them, SAD is the sum of absolute errors, is the Lagrangian operator, and B is the number of bits required for encoding. The order of CU division is as follows Figure 1 As shown in the figure, there are five ways to divide the parent CU into sub-CUs, namely quadtree division (QT), horizontal binary tree division (BH), vertical binary tree division (BV), horizontal ternary tree division (TH), and vertical ternary tree division (TV). However, in the actual division process, Figure 1 The process is performed in the order shown by the dotted line, and the prediction mode and coding information after the previous division are preserved. The image texture and motion mode in the same coding block CTU are very similar. Therefore, the information of the sub-CU of the same level (previous sub-CU) that has been divided and coded previously and the information of the parent CU of the previous level can be used to predict the possibility of the current CU selecting the affine motion estimation mode as the best mode, thereby skipping unnecessary affine motion estimation processes. At the same time, the texture and motion mode of adjacent CUs are also highly correlated, so the reconstruction information of adjacent CUs can also be used as one of the judgment conditions for skipping affine motion estimation.

[0008] The inter-frame prediction Merge mode includes a special mode called Skip mode. Skip mode is unique in that it does not transmit residuals, making it a simpler mode. Skip mode and Affine mode are mutually exclusive. Skip mode is often used in background areas, areas with less texture, and areas with slow motion. Affine motion estimation is often not the optimal mode in such areas. Therefore, the affine motion estimation process can be terminated early based on whether the optimal mode for the parent CU, preceding child CU, and adjacent CU is Skip mode, reducing the complexity of inter-frame prediction affine motion estimation. Summary of the Invention

[0009] During VVC's inter-frame prediction process, only a small number of CUs use the affine motion estimation mode as the optimal inter-frame prediction mode, and these CUs often have complex motion. However, during actual inter-frame prediction, VVC performs affine motion estimation on all CUs and selects whether to use it as the optimal prediction mode based on its rate-distortion cost. This adds many unnecessary computations, significantly increasing predictive coding time.

[0010] In view of the fact that the optimal prediction mode of a CU is correlated with its parent CU, preceding child CU, and adjacent CU during VVC inter-frame prediction, the present invention aims to propose a method for effectively reducing the complexity of inter-frame affine motion estimation without significantly reducing the efficiency of VVC compression coding. This method skips unnecessary affine motion estimation in advance by extracting the optimal prediction mode information of the current CU's parent CU, preceding child CU, and adjacent CU during VVC inter-frame prediction, thereby reducing the complexity of affine motion estimation and achieving the purpose of reducing overall encoding time without significantly affecting video bit rate and quality.

[0011] This patent proposes a method to reduce the complexity of inter-frame affine motion estimation. When performing inter-frame prediction, for the current CU, the best prediction mode of the current CU so far is first extracted, as well as the best prediction modes of the parent CU and the previous CU, and the best prediction modes of the adjacent left and upper CUs, to determine whether it is necessary to perform the affine motion estimation process at the moment, thereby skipping the affine motion estimation calculation process with a lower probability in advance. Specifically, the method includes the following steps:

[0012] (1) Perform the conventional translation motion estimation process before affine motion estimation to obtain the best inter-frame prediction mode of the parent CU, the previous child CU, the adjacent left and upper CUs, and the current best mode of the current CU;

[0013] (2) Determine whether the current best inter-frame prediction mode of the current CU is the Skip mode. If so, skip the affine motion estimation process of the current CU and execute step (6); if not, execute step (3);

[0014] (3) Determine whether there is a CU whose best inter-frame prediction mode is Skip mode in the parent CU and the preceding child CU. If so, skip the affine motion estimation process of the current CU and execute step (6); if not, execute step (4);

[0015] (4) Determine whether the optimal inter-frame prediction mode of the adjacent left and upper CUs is Skip mode. If so, skip the affine motion estimation process of the current CU and execute step (6); if not, execute step (5);

[0016] (5) performing the original affine motion estimation process;

[0017] (6) Determine the optimal inter-frame prediction mode;

[0018] (7) End the motion estimation process.

[0019] The advantages and beneficial technical effects of the present invention compared with the prior art are as follows:

[0020] (1) Compared with the traditional VVC standard coding, the fast affine motion estimation method based on reconstructed prior information proposed in the present invention significantly reduces the overall coding time of the VVC standard algorithm without significantly reducing the intra-frame coding performance;

[0021] (2) The fast affine motion estimation method based on reconstructed prior information proposed in this paper analyzes the mutual exclusivity between the VVC inter-frame Skip mode and the affine motion estimation mode, and uses this mutual exclusivity to judge the necessity of executing the affine motion estimation process in advance, skipping unnecessary calculation processes and reducing the complexity of inter-frame affine motion estimation;

[0022] (3) The fast affine motion estimation method based on reconstructed prior information proposed in this paper analyzes the correlation between the optimal prediction modes of the parent CU, the previous child CU, and the current CU between VVC frames. By utilizing this correlation, unnecessary affine motion estimation processes are skipped in advance, further saving the inter-frame affine motion estimation prediction coding time;

[0023] (4) The fast affine motion estimation method based on reconstructed prior information proposed in the present invention analyzes the correlation between the optimal inter-frame prediction modes of adjacent CUs. By utilizing this correlation, unnecessary affine motion estimation processes are skipped in advance, thereby further saving inter-frame affine motion estimation prediction coding time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is the division method of CU in VVC.

[0025] Figure 2 Flowchart of a fast affine motion estimation method based on reconstructed prior information. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to the embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Those skilled in the art may make some non-essential improvements and adjustments to the present invention based on the above invention content and implement them specifically, which should still fall within the scope of protection of the present invention.

[0027] (1) The proposed method is implemented on the VVC standard test code VTM9.1 platform. The configuration file is encoder_lowdelay_P_vtm.cfg, the GOP is 8, and the quantization parameter QP is set to 22, 27, 32, and 37.

[0028] (2) Five official standard test sequences, B, C, D, E, and F, are used to verify the coding performance of the proposed method. The specific standard test sequences used are Cactus, BQTerrace, BQMall, PartyScene, RaceHorsesC, BasketballDrill, BasketballPass, BlowingBubbles, BQSquare, RaceHorses, FourPeople, KristenAndSara, Johnny, and BasketballDrillText.

[0029] (3) In the program of the algorithm proposed in the present invention, the parameters in (1) are set for the configuration file encoder_lowdelay_P_vtm.cfg, and then all the video standard sequences to be tested are input to obtain the results of the method proposed in the present invention, and the encoding time, bit rate, and video quality are recorded and counted;

[0030] (4) In the VVC standard algorithm program, set the parameters in (1) for the configuration file encoder_lowdelay_P_vtm.cfg, then input all the video standard sequences to be tested, obtain the results of the VVC standard algorithm, and record and count the encoding time, bit rate, and video quality;

[0031] (5) The results of the above two categories are processed separately, where the coding time complexity is measured by ΔT, which represents the coding time reduction of the proposed method compared with VVC. The coding performance is measured by the objective evaluation criteria BDBR and BDPSNR;

[0032] (6) As can be seen from Table 1, compared with the VVC standard algorithm, when the coding structure is lowdelay_P, the proposed method reduces the coding time by an average of 10.02%, while the BDBR only increases by 0.24% and the BDPSNR only decreases by 0.009dB. In summary, while ensuring coding efficiency, the fast affine motion estimation algorithm proposed in this invention can effectively reduce the coding time compared to the VVC standard algorithm.

[0033] Table 1 Experimental results

[0034]

Claims

1. A fast affine motion estimation method based on reconstructed prior information, characterized by: (1) Perform the conventional translation motion estimation process before affine motion estimation to obtain the best inter-frame prediction mode of the parent CU, the previous sub-CU, the adjacent left and upper CUs, and the current best mode of the current CU. The previous sub-CU refers to the sub-CU of the same level that has been divided and encoded before; (2) Determine whether the current best inter-frame prediction mode of the current CU is the Skip mode. If so, skip the affine motion estimation process of the current CU and execute step (6); if not, execute step (3); (3) Determine whether there is a CU whose best inter-frame prediction mode is Skip mode in the parent CU and the preceding child CU. If so, skip the affine motion estimation process of the current CU and execute step (6); if not, execute step (4); (4) Determine whether the optimal inter-frame prediction mode of the adjacent left and upper CUs is Skip mode. If so, skip the affine motion estimation process of the current CU and execute step (6); if not, execute step (5); (5) performing the original affine motion estimation process; (6) Determine the optimal inter-frame prediction mode; (7) End the motion estimation process.

Images