Image anti-shake method and device, equipment and storage medium

Abstract

The invention provides an image anti-shake method and device, equipment and a storage medium. The method comprises the following steps: carrying out distortion correction on a first image to obtain a second image; determining a first rotation attitude and a first translation attitude of the second image; performing image anti-shake processing on the second image according to the first rotation attitude, the first translation attitude and angular velocity data of the IMU to obtain a third image; wherein the first image is in time synchronization with the angular velocity data.

Description

Technical Field

[0001] The present application relates to image technology, and is related to but not limited to image stabilization methods and devices, equipment, and storage media. Background Art

[0002] In recent years, with the rapid development of video shooting equipment technology, many shooting equipment have begun to support HD or even full HD shooting, such as SLR cameras, compact cameras, mobile phones with camera functions, video surveillance equipment, driving recorders, etc. Therefore, the quality of video shooting has become increasingly important for evaluating shooting equipment.

[0003] There are many factors that affect the quality of video shooting, including resolution, frame rate, noise, color saturation, image clarity, video stability, etc. Among them, video stability is a very important factor. If the shooting is unstable, the whole picture will shake, which will not only make the image unclear, but also make the human eye easily tired and the subjective experience is very poor. Especially for the currently commonly used smartphones and card cameras, since these devices are light in weight and are shot by users, it is easy to cause the picture to shake, resulting in blurred images.

[0004] The most effective solution to this type of shaking problem is to perform anti-shake processing on the captured image so that the shooting equipment has an image anti-shake function. Summary of the invention

[0005] In view of this, the image stabilization method, device, equipment, and storage medium provided in the present application improve the accuracy of the first rotational posture and the first translational posture (ie, the registration accuracy), which is beneficial to improving the image stabilization accuracy.

[0006] According to one aspect of an embodiment of the present application, an image stabilization method is provided, the method comprising: performing distortion correction on a first image to obtain a second image; determining a first rotation posture and a first translation posture of the second image; performing image stabilization processing on the second image according to the first rotation posture, the first translation posture and angular velocity data of an inertial measurement unit / inertial sensor (InertialMeasurement Unit, IMU) to obtain a third image; wherein the first image is time synchronized with the angular velocity data.

[0007] According to one aspect of an embodiment of the present application, an image stabilization device is provided, including: an anti-distortion correction module, configured to perform distortion correction on a first image to obtain a second image; a posture determination module, configured to determine a first rotation posture and a first translation posture of the second image; an image stabilization processing module, configured to perform image stabilization processing on the second image according to the first rotation posture, the first translation posture and angular velocity data of an IMU to obtain a third image; wherein the first image is time synchronized with the angular velocity data.

[0008] According to one aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0009] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0010] In an embodiment of the present application, the first rotation posture and the first translation posture used for image stabilization processing are determined based on the image after distortion correction; that is, the distortion of the first image is first removed, and then the first rotation posture and the first translation posture are determined based on the distortion-free second image, rather than determining the posture information of the image based on the first image before distortion correction, which is beneficial to improving the accuracy of the first rotation posture and the first translation posture (that is, the alignment accuracy), and further beneficial to improving the image stabilization accuracy.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0014] Figure 1It is a processing flow chart of the digital image stabilization (DIS) algorithm;

[0015] Figure 2 This is a flowchart of the Electronic Image Stabilization (EIS) algorithm processing;

[0016] Figure 3 Generate a schematic diagram for the image stabilization grid points;

[0017] Figure 4 It is the flow chart of EIS+DIS algorithm;

[0018] Figure 5 A schematic diagram of the implementation process of the image stabilization method provided in the embodiment of the present application;

[0019] Figure 6 A schematic diagram of a method implementation flow for implementing step 501 provided in an embodiment of the present application;

[0020] Figure 7 A schematic diagram of a method implementation flow for implementing step 502 provided in an embodiment of the present application;

[0021] Figure 8 A schematic diagram of a method implementation flow for implementing step 503 provided in an embodiment of the present application;

[0022] Fig. 9 A schematic diagram of a method implementation flow for implementing step 5031 provided in an embodiment of the present application;

[0023] Fig.10 A schematic diagram of the implementation process of the image stabilization method provided in the embodiment of the present application;

[0024] Fig.11 A schematic diagram of the implementation process of the image stabilization method provided in the embodiment of the present application;

[0025] Fig.12 A schematic diagram of the process principle of the anti-distortion correction module provided in the embodiment of the present application;

[0026] Fig.13 A schematic diagram of the structure of an image stabilization device provided in an embodiment of the present application;

[0027] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0030] In the following description, reference is made to “some embodiments”, “this embodiment”, “embodiments of the present application” and examples, etc., which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] The descriptions of “first, second, third” etc. that appear in the embodiments of the present application are only for illustration and distinction of the description objects. There is no distinction of order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0032] Mobile phones, tablets, smart watches and other devices usually introduce different degrees of external jitter that affects the stability of the picture due to their usage scenarios, so image stabilization is their basic requirement. There are generally two ways to implement image and video stabilization technology. One is to remove jitter through software cropping, and the other is to achieve stabilization through hardware compensation. The former can be divided into DIS based on image and video streams and EIS based on sensor streams according to the input information; the latter includes gimbal stabilization based on large mechanical equipment and OIS based on handheld micro modules.

[0033] Based on the feature matching and image alignment method and the image stabilization algorithm DIS, it mainly operates in the image domain, and estimates the inter-frame posture by feature matching. The posture is usually represented by a homography matrix; then the image stabilization is achieved by posture filtering. The process can be simplified as follows: Figure 1 .

[0034] like Figure 1 As shown, first the current frame Image N and the previous frame Image N-1Image registration is performed. The purpose of registration is to estimate the relative posture of two frames of images through feature extraction and least squares methods. Considering the possible nonlinear distortion between images, the process usually divides the image into blocks and calculates the local registration matrix; the obtained homography matrix (i.e., registration matrix) is used as the estimated posture and sent to the filtering module for filtering. In this process, posture conversion may be performed, such as the decomposition of the homography matrix, the conversion of the rotation matrix to quaternion and Euler angle; the posture sequence will be used as the motion trajectory input filter for filtering, and finally the homography matrix corresponding to the smooth posture is obtained, which is then rendered to generate a stable image.

[0035] The EIS algorithm based on sensor data takes images and sensor attitude data as input. Generally, sensor data refers to the three-axis attitude angular velocity (Gyro) and translation acceleration (Acc) output by IMU. Its processing flow can be simplified Figure 2 .

[0036] like Figure 2 As shown in the figure, the video frame and IMU sensor data are input. Since the IMU Sensor, Image Sensor and EIS system belong to different modules and have different clock signals, the frame data and IMU data need to be synchronized in time. First, the sensor data and frame data are synchronized in time. The synchronized sensor data is sent to the filter for filtering, which is the so-called attitude compensation process. The compensated attitude generates the image stabilization grid point (GridNode) for Warp. Due to the limitations of power consumption and computing power, the EIS algorithm does not perform attitude compensation calculation pixel by pixel during implementation, but rather Figure 3 As shown, the calculation is performed using grid point sampling.

[0037] The so-called Warp refers to the generation of a rotationally and translationally stabilized image using the stabilization grid points and the original image. It is a computational process of pixel interpolation based on the grid, which generally requires accelerated optimization.

[0038] The optical image stabilization system OIS based on module movement. The OIS system of mobile phones and other handheld devices generally has two-axis freedom: the lens can make a certain degree of mechanical movement in the x and y directions according to the posture and speed of the mobile phone to achieve the purpose of physical motion compensation; the mobile phone optical image stabilization system is limited by the module size and generally does not have the required image stabilization effect. It usually needs to be superimposed with EIS, which is mainly used for motion blur suppression.

[0039] The hybrid anti-shake algorithm of EIS and DIS fusion: first use the inertial sensor to track the rotation of the camera, then further estimate the camera translation by tracking the coordinate changes of feature points in the video frame, and accurately and robustly estimate the camera motion through inertial-visual fusion, and reduce the computational overhead. The general process is as follows: Figure 4shown.

[0040] The image rotation matrix R is estimated by IMU, and the image translation vector T is estimated by ORB feature matching. R and T are filtered respectively, and then an image with stable rotation and translation dimensions is obtained.

[0041] However, the image-based DIS algorithm is more dependent on image content, and motion compensation is easily affected by the content of the subject, especially large-scale motion, dark light scenes under high exposure, few overlapping content between frames, and severe motion blur, resulting in few feature points and low matching confidence. In addition, due to the distortion of the lens itself and the dynamic distortion caused by motion (water ripple effect), the linear correlation between frames is reduced, and matching must be performed in blocks to ensure accuracy, but this will result in high computational overhead and poor real-time performance.

[0042] The EIS algorithm system performs anti-shake calculation based on sensor data, and its effect depends on the accuracy, range and sampling frequency of the sensor. Under normal circumstances, due to the process and production line calibration of IMU sensors, the data often has zero drift and temperature drift errors. This error is reflected in the process of integrating gyro data into attitude data, which will produce a certain cumulative error. On the one hand, this error can be very large, and in severe cases it may reach more than several degrees per second, which will have a serious impact on the anti-shake effect. On the other hand, this error is a dynamic error and cannot be determined by zero bias calibration. In addition, although IMU records acceleration information at the same time, this information cannot be used directly in the pixel coordinate system and has a high instantaneous error. Therefore, the EIS algorithm based on IMU can only use gyro data for rotational anti-shake, and cannot process the translation. Moreover, because EIS needs to crop the picture, it will lose image quality and FOV, and at the same time, it will amplify the perception of motion blur for dark light scenes. It usually reduces the perception of motion blur by suppressing the anti-shake intensity. In addition, since the attitude sensor usually belongs to a different clock system from the image sensor, there is a delay in the data, which has a greater impact on high-frequency motion scenes.

[0043] Due to the limitations of module technology and size, the OIS system generally has only two degrees of freedom in terms of dimensionality, and cannot fully compensate for the three-axis rotation of real scenes. On the other hand, its compensation range is very small, generally not exceeding 2° in the x and y directions, which cannot cover the needs of most shooting scenes.

[0044] The EIS+DIS fusion anti-shake system generally hopes to achieve the purpose of complementary advantages by fusing EIS based on sensor data and DIS based on image methods. However, most of the current EIS+DIS fusion systems have two problems. First, they do not consider the impact of lens distortion and motion distortion on the DIS motion estimation process, and usually do not provide a method to remove global nonlinear distortion, which will seriously affect the registration accuracy. Second, there is a lack of investigation on the zero bias error and temperature drift error of the IMU sensor. In general, a light coupling method is adopted in which the IMU is responsible for rotation and the image is responsible for translation. However, in fact, the accumulated error of the IMU data will also cause problems with the rotation matrix R. Therefore, only IMU data cannot be used for rotation estimation.

[0045] Based on this, an embodiment of the present application provides an image stabilization method. Figure 5 A schematic diagram of the implementation process of the image anti-shake method provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the method may include the following steps 501 to 503:

[0046] Step 501, performing distortion correction on the first image to obtain a second image;

[0047] Step 502, determining a first rotation posture and a first translation posture of the second image;

[0048] Step 503: Perform image stabilization processing on the second image according to the first rotation posture, the first translation posture and the angular velocity data of the IMU to obtain a third image; wherein the first image is time synchronized with the angular velocity data.

[0049] In the embodiment of the present application, the first rotation posture and the first translation posture for image stabilization processing are determined based on the image after distortion correction; that is, the distortion of the first image is first removed, and then the first rotation posture and the first translation posture are determined based on the second image without distortion, rather than determining the posture information of the image based on the first image before distortion correction, thereby helping to improve the accuracy of the first rotation posture and the first translation posture (that is, the registration accuracy), and further helping to improve the image stabilization accuracy. On the other hand, because the distortion is removed before registration, there is no need to perform block segmentation on the image during registration.

[0050] The following describes further optional implementations and related terms of each of the above steps.

[0051] In step 501, distortion correction is performed on the first image to obtain a second image.

[0052] In some embodiments, the first image may be image data output by an image sensor. It is understood that the distortion in an image generally includes two types: one is the inherent distortion caused by the module lens, which is fixed and is called camera distortion; the other is the water ripple distortion (generally referred to as the jelly effect) caused by camera motion and the rolling shutter effect of the CMOS image sensor, which is caused by the motion state and is called motion distortion. In step 501, the distortion correction of the first image includes correction of camera distortion and motion distortion.

[0053] In some embodiments, Figure 6 As shown, step 501 can be implemented through the following steps 5011 to 5013:

[0054] Step 5011: back-project the first sample pixel coordinates of the first image into a three-dimensional coordinate system according to the camera intrinsic parameters and the first camera distortion parameters to obtain second sample pixel coordinates.

[0055] In some embodiments, the three-dimensional coordinate system may be a world coordinate system. In a possible implementation, a mapping relationship table between the pixel coordinates of the image and the three-dimensional coordinate system may be pre-established, based on which the second sample pixel coordinates are obtained by looking up the table.

[0056] In the embodiment of the present application, whether all pixel coordinates in the first image are back-projected into the three-dimensional coordinate system is not limited. It can be that part of the pixel coordinates or all of the pixel coordinates are back-projected into the three-dimensional coordinate system. Part of the pixel coordinates can be pixel coordinates sampled from the first image according to the sampling rule.

[0057] For ease of understanding, step 5011 can be expressed as the following formula (1):

[0058] P i =F -1 (p i ;K,r) (1)

[0059] Wherein, P is the second sample pixel coordinate, p is the pixel point coordinate in the image coordinate system (i.e., the first sample pixel coordinate); F -1 is the back-projection function, K and r are projection parameters, K is the camera intrinsic parameter, and r is the camera distortion parameter;

[0060] In this way, the camera distortion in the first image is removed by back-projection.

[0061] Step 5012: Perform motion distortion correction on the second sample pixel coordinates according to the angular velocity data to obtain third sample pixel coordinates.

[0062] Specifically, in some embodiments, motion distortion correction can be implemented as follows: determining the rotational posture of an exposure sequence of a single exposure of the first image based on the angular velocity data; wherein the exposure moments of different exposure sequences are different; determining a rotational posture difference between a non-reference exposure sequence in the first image and a reference exposure sequence of the first image; and performing posture correction on a second sample pixel coordinate belonging to the corresponding non-reference exposure sequence based on the rotational posture difference to obtain the third sample pixel coordinate.

[0063] It can be understood that the so-called exposure sequence of a single exposure can be understood as the image data output by the image sensor each time it is exposed, for example, the image sensor outputs a line of exposure sequence / image sequence each time it is exposed. Then, in a possible implementation, the angular velocity data collected / measured by the IMU at the same moment can be obtained according to the exposure moment (or exposure end moment) of the exposure sequence of the first image, and the angular velocity data at the same moment can be used as the rotational posture of the exposure sequence. Of course, in practical applications, the angular velocity data measured by the IMU may not have a measurement timestamp aligned with the exposure moment (or exposure end moment) of the exposure sequence, so the angular velocity data closest to the exposure moment (or exposure end moment) of the exposure sequence can also be found from the angular velocity data measured by the IMU, and it can be used as the rotational posture of the exposure sequence, or the corresponding angular velocity data can be interpolated according to the exposure moment (or exposure end moment), and it can be used as the rotational posture of the exposure sequence.

[0064] In the embodiment of the present application, it is not limited to which exposure sequence in the first image is used as the reference exposure sequence, and any exposure sequence in the first image can be used as the reference exposure sequence. For example, in some embodiments, the first row or the middle row in the first image is used as the reference exposure sequence, and the remaining rows are non-reference exposure sequences.

[0065] Assume that the rotational posture difference between a non-reference exposure sequence and the reference exposure sequence of the first image is expressed as Q diff , then the second sample pixel coordinate P belonging to the corresponding non-reference exposure sequence can be obtained according to the following formula (2): i Perform posture correction to obtain the third sample pixel coordinate P′ i :

[0066] P′ i =Q diff P i Q diff * (2)

[0067] In this way, motion distortion in the image can be removed.

[0068] Step 5013: Render the first image according to the third sample pixel coordinates to obtain the second image.

[0069] Further, in some embodiments, the second image can be obtained as follows: according to the camera intrinsic parameter and the second camera distortion parameter, the third sample pixel coordinate is projected to the image coordinate system to obtain a fourth sample pixel coordinate; according to the fourth sample pixel coordinates respectively corresponding to a plurality of first sample pixel coordinates of the first image, an anti-distortion grid of the first image is determined; according to the anti-distortion grid, the first image is rendered to obtain the second image.

[0070] For ease of understanding, the fourth sample pixel coordinate p′ can be obtained by the following formula (3):

[0071] p′=F(P′ i ;K,r′) (3)

[0072] Among them, F is the projection function, K is the camera internal parameter, r′ is the second camera distortion parameter, P′ i is the third sample pixel coordinate. Exemplarily, the second camera distortion parameter is equal to 0.

[0073] In some embodiments, an appropriate scale transformation may be performed during the projection process to obtain an anti-distortion grid that meets the image size.

[0074] It can be understood that the second image obtained through steps 5011 to 5013 is the result of removing camera distortion and motion distortion; this is beneficial to improving the image stabilization accuracy.

[0075] In step 502, a first rotation posture and a first translation posture of the second image are determined.

[0076] In a possible implementation, the posture information of the second image may be determined by an image registration method. Specifically, in some embodiments, Figure 7 As shown, step 502 can be implemented through the following steps 5021 to 5023:

[0077] Step 5021, obtaining a stored fourth image after the distortion correction is performed on the initial frame image.

[0078] The image after distortion correction can be called an undistorted image, and the fourth image is the first undistorted image in the historical undistorted images stored in the storage module. The timestamp of the first undistorted image, that is, the fourth image, is aligned with the timestamp of the initial angular velocity data of the IMU, which is beneficial to improve the fusion accuracy of the two rotational postures mentioned in step 5031 below, thereby improving the image stabilization accuracy.

[0079] Step 5022: register the second image with the fourth image to obtain a registration matrix.

[0080] In the embodiments of the present application, there is no limitation on the image feature extraction algorithm used in image registration. Different image feature extraction algorithms can be selected based on actual conditions during registration, such as SIFT, SURF, FAST or optical flow method. Based on the optimization method, the least square solution of the feature point set registration error is solved to obtain the registration matrix.

[0081] Step 5023: Determine the first rotation posture and the first translation posture according to the registration matrix.

[0082] In a possible implementation manner, SVD decomposition may be performed on the registration matrix (ie, the homography matrix) to obtain the first rotation posture and the first translation posture.

[0083] It can be understood that in the embodiments of the present application,

[0084] In step 503, image stabilization processing is performed on the second image according to the first rotation posture, the first translation posture and the angular velocity data of the IMU to obtain a third image; wherein the first image is time synchronized with the angular velocity data.

[0085] In some embodiments, Figure 8 As shown, step 503 can be implemented by the following steps 5031 and 5032:

[0086] Step 5031: Fusing the first rotation posture with the angular velocity data to obtain a second rotation posture.

[0087] It can be understood that before performing image stabilization based on the first rotation posture, the first rotation posture and the angular velocity data of the IMU are first fused, and then image stabilization is performed based on the fused second rotation posture; in this way, the advantages of digital image stabilization and electronic image stabilization complement each other, which is beneficial to improving the image stabilization accuracy.

[0088] In some embodiments, Fig. 9 As shown, the fusion of the first rotation posture and the angular velocity data can be achieved through the following steps 901 to 904:

[0089] Step 901, reprojecting the second image according to the registration matrix to obtain a fifth image;

[0090] Step 902, determining a reprojection error according to the fifth image and the fourth image;

[0091] Step 903: Determine the registration confidence of the registration matrix according to the reprojection error.

[0092] Exemplarily, in some embodiments, the registration confidence c may be determined according to the following formula (4):

[0093]

[0094] Where e is the reprojection error.

[0095] Step 904: According to the registration confidence, fuse the first rotation posture and the angular velocity data to obtain the second rotation posture.

[0096] There is no limitation on the fusion algorithm of step 904. By way of example, in some embodiments, step 904 may be implemented as follows: converting the first rotational posture into a first quaternion; converting the angular velocity data into a second quaternion; using the alignment confidence as an interpolation coefficient, performing spherical linear interpolation between the first quaternion and the second quaternion to obtain the second rotational posture.

[0097] That is, the second rotation posture Q is obtained according to the following formula (5):

[0098] Q = slerp(Q img ,Q imu ,c) (5)

[0099] Among them, Q img is the first quaternion, Q img is the second quaternion. Slerp is a spherical interpolation method, which is similar to the sliding average of the following formula (6):

[0100] a=θ*b+(1-θ)*c (6)

[0101] Among them, θ can be understood as the registration confidence c, a can be understood as Q, and b can be understood as Q img , c can be understood as Q imu .

[0102] It can be understood that in the embodiment described in steps 901 to 904, the fusion is not a simple fusion, but a fusion of two types of data based on the registration confidence; this is beneficial to improving the accuracy of the fusion, and thus is beneficial to improving the image stabilization accuracy.

[0103] Step 5032: Perform image stabilization processing on the second image according to the second rotation posture and the first translation posture to obtain a third image.

[0104] Further, in some embodiments, step 5032 can be implemented as follows: using the alignment confidence as a filtering coefficient, filtering the second rotation posture and the first translation posture respectively to obtain a reference rotation posture and a reference translation posture; determining the anti-shake rotation amount based on the reference rotation posture and the second rotation posture; determining the anti-shake translation amount based on the reference translation posture and the first translation posture; rendering the second image based on the anti-shake rotation amount and the anti-shake translation amount to obtain the third image.

[0105] It can be understood that in the embodiment of the present application, the filter coefficient is the alignment confidence rather than a fixed preset value; this is beneficial to improving the accuracy of the anti-shake rotation amount and the anti-shake translation amount, thereby helping to improve the image anti-shake accuracy.

[0106] For example, in some embodiments, the anti-shake rotation amount and the anti-shake translation amount may be determined according to the following equations (7) and (8):

[0107]

[0108] Δt=t ref -t (8)

[0109] Among them, ΔQ represents the anti-shake rotation amount, Δt represents the anti-shake translation amount, Q ref represents the reference rotation posture, Q represents the second rotation posture, t ref represents the reference translation posture, and t represents the first translation posture.

[0110] Note that t ref –t instead of t–t ref The reason is that the final adjustment is the coordinate system adjustment rather than the projection point adjustment. The same is true for Q.

[0111] The present application embodiment further provides an image anti-shake method, Fig.10 A schematic diagram of the implementation process of the image anti-shake method provided in the embodiment of the present application is shown in FIG. Fig.10 As shown, the method includes the following steps 1001 to 1020:

[0112] Step 1001, back-projecting first sample pixel coordinates of the first image into a three-dimensional coordinate system according to camera intrinsic parameters and first camera distortion parameters to obtain second sample pixel coordinates;

[0113] Step 1002, determining the rotational posture of an exposure sequence of a single exposure of the first image according to the angular velocity data of the IMU; wherein the exposure times of different exposure sequences are different;

[0114] Step 1003, determining a rotational posture difference between a non-reference exposure sequence in the first image and a reference exposure sequence in the first image;

[0115] Step 1004 : performing posture correction on the second sample pixel coordinates belonging to the corresponding non-reference exposure sequence according to the rotation posture difference to obtain the third sample pixel coordinates.

[0116] Step 1005, projecting the third sample pixel coordinates to an image coordinate system according to the camera intrinsic parameters and the second camera distortion parameters to obtain fourth sample pixel coordinates;

[0117] Step 1006, determining an anti-distortion grid of the first image according to the fourth sample pixel coordinates respectively corresponding to the plurality of first sample pixel coordinates of the first image;

[0118] Step 1007: Render the first image according to the anti-distortion grid to obtain the second image.

[0119] Step 1008, obtaining a stored fourth image after the distortion correction is performed on the initial frame image;

[0120] Step 1009, registering the second image with the fourth image to obtain a registration matrix;

[0121] Step 1010: Determine the first rotation posture and the first translation posture according to the registration matrix.

[0122] Step 1011, reprojecting the second image according to the registration matrix to obtain a fifth image;

[0123] Step 1012, determining a reprojection error according to the fifth image and the fourth image;

[0124] Step 1013, determining the registration confidence of the registration matrix according to the reprojection error;

[0125] Step 1014, converting the first rotation posture into a first quaternion;

[0126] Step 1015, converting the angular velocity data into a second quaternion;

[0127] Step 1016: Using the registration confidence as an interpolation coefficient, perform spherical linear interpolation between the first rotational posture and the angular velocity data to obtain the second rotational posture.

[0128] Step 1017, using the registration confidence as a filter coefficient, filtering the second rotation posture and the first translation posture respectively to obtain a reference rotation posture and a reference translation posture;

[0129] Step 1018, determining an anti-shake rotation amount according to the reference rotation posture and the second rotation posture;

[0130] Step 1019, determining an anti-shake translation amount according to the reference translation posture and the first translation posture;

[0131] Step 1020: Render the second image according to the anti-shake rotation amount and the anti-shake translation amount to obtain the third image.

[0132] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0133] In an embodiment of the present application, a hybrid video image stabilization method based on IMU sensor data and image data is provided, in which nonlinear distortion of the image is removed by IMU sensor data, and the inter-frame posture change of the video is obtained by using the distortion-free image, and the image posture is fused with the IMU sensor posture for hybrid stabilization. The hybrid stabilization of image and IMU sensors can assist each other in improving the stabilization effect: image information can achieve translation stabilization that IMU data does not have, and when the confidence level is sufficient and there is no IMU posture drift, IMU data can handle complex scenarios when the image information fails to match due to external interference.

[0134] In the embodiment of the present application, an EIS+OIS fusion system algorithm model is provided, namely, a hybrid anti-shake algorithm system model: compared with the traditional EIS anti-shake algorithm based on IMU sensors and the DIS anti-shake algorithm based on image information, this algorithm model realizes the complementary fusion of the two anti-shake strategies, and has the advantages of both methods and avoids the original disadvantages through data fusion. At the same time, before fusion, this method first corrects the nonlinear distortion of the image, which is more robust to the fusion error introduced by motion distortion than the existing direct fusion strategy.

[0135] like Fig.11 As shown in the figure, it is a block diagram of the hybrid anti-shake algorithm system. The entire system can be divided into three modules: nonlinear distortion correction module, attitude analysis and fusion module, and comprehensive filtering module.

[0136] Nonlinear distortion correction module, responsible for correcting lens distortion and water ripples caused by motion;

[0137] The attitude analysis and fusion module is responsible for analyzing the IMU sensor data and image data and fusing the attitude trajectory;

[0138] The comprehensive filtering module is responsible for smoothing filtering based on the fused posture trajectory and obtaining the reference posture of the smooth trajectory.

[0139] Regarding the correction of nonlinear distortion, its principle and process are as follows Fig.12 shown.

[0140] The nonlinear distortion described in the embodiment of the present application refers to the distortion that is non-uniformly distributed in the image. The degree of distortion is related to the spatial position of the image and does not have global consistency. There are two types of distortion in the video stream. One is the inherent distortion caused by the module lens. This distortion is fixed and determined by the lens design and production process. It is camera distortion and can generally be determined by offline camera calibration. The other is the water ripple distortion (generally referred to as the jelly effect) caused by camera motion and CMOS rolling shutter effect. This distortion is caused by the motion state and is dynamic distortion / motion distortion. When processing, the nonlinear distortion correction module first converts the two-dimensional ( 2D) image points are projected into a three-dimensional (3D) space without lens distortion to obtain a 3D image, in which lens distortion (also known as camera distortion) is corrected during projection; then in the 3D space, the Gyro data at the same time is indexed from the IMU sensor based on the exposure time of each row of the image sensor and integrated into a quaternion attitude; then a row of attitudes in the 3D image (usually the first row or the middle row) is selected as the reference baseline attitude Q_base, and the attitudes of other rows are uniformly corrected to Q_base; finally, the corrected 3D image points are projected back to the 2D space using the projection function F to obtain a distortion-free mesh (also known as an anti-distortion mesh).

[0141] The specific process steps of the complete hybrid anti-shake algorithm are as follows 1 to 13:

[0142] 1. The Gyro data output by the IMU sensor and the offline calibrated lens intrinsic parameters (i.e., camera intrinsic parameters) and camera distortion parameters are used as inputs of the anti-distortion correction module; wherein the Gyro data can be understood as the angular velocity data;

[0143] 2. As shown in the following formula (9), the 2D image coordinate system is back-projected into the 3D space based on the lens intrinsic parameters and the camera distortion parameters, and the lens distortion is removed in this process:

[0144] P i =F -1 (p i ;K,r) (9)

[0145] Wherein, P is the projection point in 3D space (i.e., the second sample pixel coordinate), p is the pixel point coordinate in the image coordinate system (the first sample pixel coordinate); F -1 is the back-projection function, which transforms the pixel coordinates in the 2D image space into the 3D coordinate system. K and r are projection parameters, K is the lens intrinsic parameter, and r is the camera distortion parameter.

[0146] 3. In the 3D coordinate system, the rotational posture of each row of the image is obtained based on the Gyro data of the IMU sensor. The first row is taken as the base row (Q_base) to calculate the posture difference Q between the remaining rows and the first row. diff (All postures are expressed using quaternions), as shown in the following formula (10) using Q diff Adjust the postures of the remaining rows to align with the base row to remove the water ripple distortion caused by movement:

[0147] P′ i =Q diff P i Q diff * (10)

[0148] 4. The adjusted 3D space point is projected back to the 2D coordinate system through the projection function F as shown in the following formula (11). In this process, the camera distortion parameter r=0:

[0149] p′=F(P′ i ;K,r′) (11)

[0150] p′ can be understood as the fourth sample pixel coordinate;

[0151] 5. After appropriate scale transformation during the projection process, the anti-distortion mesh (i.e. anti-distortion mesh) that meets the output image size is output as the input of the rendering module;

[0152] 6. Image N (i.e. the first image) and the anti-distortion mesh are input into the rendering module to generate an undistorted image N -undistort (i.e. the second image);

[0153] 7. Image N -undistort is registered with the first frame of the undistorted image image0-undistort (i.e., the fourth image) stored in the historical undistorted image storage module. Different image feature extraction algorithms can be selected based on actual conditions during registration, such as SIFT, SURF, FAST, or optical flow. Based on the optimization method, the least square solution of the registration error of the feature point set is obtained as shown in the following formula (12), and the registration matrix H is calculated:

[0154]

[0155] Where p i Represents the matched feature point set;

[0156] 8. Reprojection is performed based on the solved registration matrix H. The global reprojection error e is calculated as shown in equation (13), and the registration confidence c is calculated based on the reprojection error (as shown in equation (14)):

[0157]

[0158]

[0159] 9. Perform SVD decomposition on the registration matrix H (homography matrix) to obtain the rotation matrix R (i.e., the first rotation posture) and the translation vector t (i.e., the first translation posture), where the rotation matrix can be converted into a quaternion Q img express:

[0160] H=K(R img |t img )K -1 (15)

[0161] R img ~Q img (16)

[0162] Where K is the camera intrinsic parameter, R is the rotation matrix, Q is the quaternion converted from the rotation matrix, and t is the translation vector;

[0163] 10. Image-based N The exposure end timestamp is obtained by searching the gyro data with the closest timestamp in the IMU's gyro data, or interpolating the corresponding gyro data based on the timestamp, and integrating the data into the quaternion Q imu ; Before that, the image sensor and IMU sensor need to be clock synchronized;

[0164] 11. The quaternion Q of the Gyro data of the IMU is calculated based on the registration confidence c as follows (17): imu and image registration quaternion Q img To perform the fusion:

[0165] Q = slerp(Q img ,Q imu ,c) (17)

[0166] Where Q can be understood as the second rotation attitude, slerp is a spherical interpolation method, which is often used for attitude interpolation, and the registration confidence c is the interpolation coefficient (between 0 and 1), which has an effect similar to the sliding average of the following equation (18):

[0167] a=θ*b+(1-θ)*c (18)

[0168] Among them, θ can be understood as the registration confidence c, a can be understood as Q, and b can be understood as Q img , c can be understood as Q imu .

[0169] However, slerp is interpolation on the sphere, not linear interpolation. The significance of IMU and image attitude fusion is that since image information is affected by image content during registration, when large-scale violent movement or long exposure in a dark environment causes violent motion blur, the registration rotation matrix R error will be large. If image attitude is used for anti-shake at this time, it will affect the effect. As for the IMU sensor attitude, since it measures the attitude of the IMU element rather than directly measuring the attitude of the image sensor, there may be a relative attitude between the two, which is introduced by the assembly error. At the same time, since the IMU sensor itself has zero drift and temperature drift errors, the accumulation will cause attitude drift. Moreover, since the image sensor uses different clock sources, there may be unstable time errors between the data of the two. Therefore, it is necessary to fuse the IMU attitude and image registration attitude, and dynamically adjust the fusion ratio of IMU and image attitude through registration confidence to cope with different video shooting environments.

[0170] 12. The fused mixed rotation posture Q, the separated translation vector t and the registration confidence c are used as the input of the comprehensive filtering module. Q and t are filtered respectively. Considering the factor that the translation vector may be disturbed, the registration confidence c is used as the input of the translation filtering module. After filtering, the reference rotation posture Q is output. ref and the reference translation pose t ref , the anti-shake rotation amount ΔQ and the anti-shake translation amount Δt are solved based on the reference posture using the following equations (19) and (20):

[0171]

[0172] Δt=t ref -t (20)

[0173] Note that t ref –t instead of t–t ref , because the final adjustment is the coordinate system adjustment rather than the projection point adjustment, and the same is true for Q;

[0174] 14. Image N -undistort, ΔQ and Δt are used as input to the rendering module to image N -undistort for rendering and finally generate image N -stable (i.e. the third image).

[0175] In the embodiment of the present application, a new hybrid anti-shake system strategy of IMU sensor and image information fusion is proposed, which unifies the original discrete IMU-based EIS and image-based DIS into the same system; the new system integrates the advantages of the EIS and DIS methods, and complements the disadvantages of the two through data fusion. The integrated hybrid anti-shake system is mainly divided into three parts: nonlinear distortion correction module, attitude analysis fusion module and comprehensive filtering module; among them, the nonlinear distortion correction module is responsible for removing the dynamic distortion (water ripples) caused by the fixed distortion of the lens and the motion, the attitude analysis fusion module is responsible for fusing the IMU attitude and the image registration attitude; the comprehensive filtering module is responsible for filtering and generating a jitter-free reference attitude. Through distortion correction and data fusion, on the one hand, the distortion correction and attitude correction are decoupled, so that the global image registration attitude can represent the current image attitude, reducing the amount of calculation increased by block registration; on the other hand, the IMU sensor data does not rely on the image content to represent the attitude and the image registration attitude takes into account the rotation and translation information. These two advantages can be integrated, thereby improving the comprehensive anti-shake effect.

[0176] In the embodiments of the present application,

[0177] 1) A new hybrid anti-shake solution combining EIS and DIS is proposed;

[0178] 2) The new EIS+DIS hybrid anti-shake system decouples the distortion correction and attitude filtering processes, improving the accuracy of image attitude estimation;

[0179] 3) The hybrid anti-shake system adopts an IMU attitude and image attitude fusion solution based on image registration confidence, which can have better scene adaptability;

[0180] 4) Hybrid stabilization system can take into account both rotational stabilization and translational stabilization;

[0181] It should be noted that

[0182] 1) The hybrid anti-shake system proposed in the embodiment of the present application is a complete fusion system, which can be regarded as a superposition of a software anti-shake system and an optical anti-shake OIS system;

[0183] 2) The image registration matrix calculation mentioned in the embodiments of the present application does not involve a specific feature extraction algorithm. When implementing it, an algorithm solution needs to be selected according to the specific situation of the hardware structure;

[0184] 3) The EIS or DIS filtering module mentioned in the embodiments of the present application does not involve a specific algorithm model, but is an algorithm system model framework. When it is implemented, the algorithm solution needs to be selected according to the specific situation of the hardware structure;

[0185] 4) The time synchronization method mentioned in the embodiments of the present application can achieve clock-level synchronization by changing the hardware connection, or can achieve software-level synchronization by software calibration.

[0186] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.; or, steps in different embodiments may be combined into a new technical solution.

[0187] Based on the above embodiments, the present application provides an image stabilization device. Fig.13 A schematic diagram of the structure of the image stabilization device provided in the embodiment of the present application is shown in FIG. Fig.13 As shown, the image anti-shake device 13 includes:

[0188] The anti-distortion correction module 131 is configured to perform distortion correction on the first image to obtain a second image;

[0189] a posture determination module 132, configured to determine a first rotation posture and a first translation posture of the second image;

[0190] The image stabilization processing module 133 is configured to perform image stabilization processing on the second image according to the first rotation posture, the first translation posture and the angular velocity data of the IMU to obtain a third image; wherein the first image is time synchronized with the angular velocity data.

[0191] In some embodiments, the image stabilization processing module 133 is configured to: fuse the first rotational posture with the angular velocity data to obtain a second rotational posture; and perform image stabilization processing on the second image according to the second rotational posture and the first translational posture to obtain a third image.

[0192] In some embodiments, the anti-distortion correction module 131 is configured to: back-project the first sample pixel coordinates of the first image into a three-dimensional coordinate system according to the camera intrinsic parameters and the first camera distortion parameters to obtain second sample pixel coordinates; perform motion distortion correction on the second sample pixel coordinates according to the angular velocity data to obtain third sample pixel coordinates; and render the first image according to the third sample pixel coordinates to obtain the second image.

[0193] In some embodiments, the anti-distortion correction module 131 is configured to: determine the rotational posture of the exposure sequence of the single exposure of the first image according to the angular velocity data; wherein the exposure moments of different exposure sequences are different; determine the rotational posture difference between the non-reference exposure sequence in the first image and the reference exposure sequence of the first image; and perform posture correction on the second sample pixel coordinates belonging to the corresponding non-reference exposure sequence according to the rotational posture difference to obtain the third sample pixel coordinates.

[0194] In some embodiments, the anti-distortion correction module 131 is configured to: project the third sample pixel coordinates to the image coordinate system according to the camera intrinsic parameters and the second camera distortion parameters to obtain fourth sample pixel coordinates; determine the anti-distortion grid of the first image according to the fourth sample pixel coordinates respectively corresponding to multiple first sample pixel coordinates of the first image; and render the first image according to the anti-distortion grid to obtain the second image.

[0195] In some embodiments, the posture determination module 132 is configured to: obtain the stored fourth image after the distortion correction is performed on the initial frame image; align the second image with the fourth image to obtain a registration matrix; and determine the first rotation posture and the first translation posture according to the registration matrix.

[0196] In some embodiments, the image stabilization processing module 133 is configured to: reproject the second image according to the registration matrix to obtain a fifth image; determine the reprojection error according to the fifth image and the fourth image; determine the registration confidence of the registration matrix according to the reprojection error; and fuse the first rotation posture and the angular velocity data according to the registration confidence to obtain the second rotation posture.

[0197] In some embodiments, the image stabilization processing module 133 is configured to: convert the first rotational posture into a first quaternion; convert the angular velocity data into a second quaternion; use the alignment confidence as an interpolation coefficient, perform spherical linear interpolation between the first quaternion and the second quaternion, and obtain the second rotational posture.

[0198] In some embodiments, the image stabilization processing module 133 is configured to: use the alignment confidence as a filter coefficient, filter the second rotation posture and the first translation posture respectively, and obtain a reference rotation posture and a reference translation posture; determine the stabilization rotation amount according to the reference rotation posture and the second rotation posture; determine the stabilization translation amount according to the reference translation posture and the first translation posture; render the second image according to the stabilization rotation amount and the stabilization translation amount to obtain the third image.

[0199] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0200] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0201] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0202] An embodiment of the present application provides an electronic device, Fig.14 Schematic diagram of the hardware entity of the electronic device of the embodiment of the present application, such as Fig.14 As shown, the electronic device 14 includes a memory 141 and a processor 142. The memory 141 stores a computer program that can be run on the processor 142. When the processor 142 executes the program, the steps in the method provided in the above embodiment are implemented.

[0203] It should be noted that the memory 141 is configured to store instructions and applications executable by the processor 142, and can also cache data to be processed or processed by the processor 142 and various modules in the electronic device 14 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0204] During the implementation process, the electronic device 14 may be various types of devices with image processing capabilities, for example, the electronic device may include a mobile phone, a tablet computer, a desktop computer, a personal digital assistant, a navigator, a digital phone, a video phone, a television or a sensor device.

[0205] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0206] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0207] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0208] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0209] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0210] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0211] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0212] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0213] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0214] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0215] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0216] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain a new method embodiment. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain a new product embodiment. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain a new method embodiment or device embodiment.

[0217] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image stabilization method, characterized in that: The method comprises: Performing distortion correction on the first image to obtain a second image; determining a first rotational pose and a first translational pose of the second image; According to the first rotation posture, the first translation posture and the angular velocity data of the IMU, image stabilization processing is performed on the second image to obtain a third image; wherein the first image is time synchronized with the angular velocity data.

2. The method according to claim 1, characterized in that The step of performing image stabilization processing on the second image according to the first rotation posture, the first translation posture and the angular velocity data of the IMU to obtain a third image includes: Fusing the first rotational posture with the angular velocity data to obtain a second rotational posture; According to the second rotation posture and the first translation posture, image stabilization processing is performed on the second image to obtain a third image.

3. The method according to claim 1, characterized in that The step of performing distortion correction on the first image to obtain the second image includes: Back-projecting the first sample pixel coordinates of the first image into a three-dimensional coordinate system according to the camera intrinsic parameters and the first camera distortion parameter to obtain second sample pixel coordinates; According to the angular velocity data, motion distortion correction is performed on the second sample pixel coordinates to obtain third sample pixel coordinates; The first image is rendered according to the third sample pixel coordinates to obtain the second image.

4. The method according to claim 3, characterized in that The step of performing motion distortion correction on the second sample pixel coordinates according to the angular velocity data to obtain third sample pixel coordinates includes: Determining, according to the angular velocity data, a rotational posture of an exposure sequence of a single exposure of the first image; wherein exposure moments of different exposure sequences are different; determining a rotational attitude difference between a non-reference exposure sequence in the first image and a reference exposure sequence in the first image; According to the rotational attitude difference, attitude correction is performed on the second sample pixel coordinates belonging to the corresponding non-reference exposure sequence to obtain the third sample pixel coordinates.

5. The method according to claim 3 or 4, characterized in that: The rendering of the first image according to the third sample pixel coordinates to obtain the second image includes: Projecting the third sample pixel coordinates to an image coordinate system according to the camera intrinsic parameters and the second camera distortion parameters to obtain fourth sample pixel coordinates; determining an anti-distortion grid of the first image according to the fourth sample pixel coordinates respectively corresponding to the plurality of first sample pixel coordinates of the first image; The first image is rendered according to the anti-distortion grid to obtain the second image.

6. The method according to any one of claims 2 to 5, characterized in that: The determining a first rotation posture and a first translation posture of the second image comprises: Acquire a stored fourth image after the distortion correction is performed on the initial frame image; Registering the second image with the fourth image to obtain a registration matrix; The first rotational posture and the first translational posture are determined according to the registration matrix.

7. The method according to claim 6, characterized in that The step of fusing the first rotational posture with the angular velocity data to obtain a second rotational posture includes: reprojecting the second image according to the registration matrix to obtain a fifth image; determining a reprojection error based on the fifth image and the fourth image; Determining a registration confidence of the registration matrix according to the reprojection error; The first rotation posture and the angular velocity data are fused according to the registration confidence to obtain the second rotation posture.

8. The method according to claim 7, characterized in that The fusing the first rotational posture and the angular velocity data according to the registration confidence to obtain the second rotational posture includes: Converting the first rotational posture into a first quaternion; Converting the angular velocity data into a second quaternion; The registration confidence is used as an interpolation coefficient, and spherical linear interpolation is performed between the first quaternion and the second quaternion to obtain the second rotation posture.

9. The method according to claim 7, characterized in that: The step of performing image stabilization processing on the second image according to the second rotation posture and the first translation posture to obtain a third image includes: Using the registration confidence as a filter coefficient, filtering the second rotation posture and the first translation posture respectively to obtain a reference rotation posture and a reference translation posture; determining an anti-shake rotation amount according to the reference rotation posture and the second rotation posture; Determining an anti-shake translation amount according to the reference translation posture and the first translation posture; The second image is rendered according to the anti-shake rotation amount and the anti-shake translation amount to obtain the third image.

10. An image stabilization device, characterized in that: include: an anti-distortion correction module, configured to perform distortion correction on the first image to obtain a second image; a posture determination module configured to determine a first rotation posture and a first translation posture of the second image; The image stabilization processing module is configured to perform image stabilization processing on the second image according to the first rotation posture, the first translation posture and the angular velocity data of the IMU to obtain a third image; wherein the first image is time synchronized with the angular velocity data.

11. An electronic device comprising a first memory and a first processor, wherein the first memory stores a computer program that can be run on the first processor, characterized in that: When the first processor executes the program, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images