IMU (Inertial Measurement Unit)-based real-time video image stabilization method and system for intra-frame motion compensation

Abstract

The invention provides a real-time video image stabilization method and system for performing intra-frame motion compensation based on an IMU (Inertial Measurement Unit). The method comprises the following steps: S1, preprocessing data; s2, track smoothing processing is carried out; s3, performing intra-frame motion compensation: S3.1, initializing line exposure delay time; s3.2, calculating the time difference between the current vth line and the middle line exposure in the current frame image; s3.3, acquiring the smoothed virtual camera pose of the current frame and the real camera pose of each layer; s3.4, acquiring a motion compensation matrix for compensating the line of image to the virtual pose of the frame of image; s3.5, acquiring an image frame perspective transformation matrix according to the camera internal reference matrix K and the motion compensation matrix; s3.6, performing intra-frame motion compensation on the image layers to obtain an initial motion compensation matrix queue; s3.7, carrying out black edge removal processing on the motion compensation matrix queue; obtaining a motion compensation matrix queue after black edge removal; and S4, image transformation output. The system comprises a data preprocessing module, a track smoothing module, an intra-frame motion compensation module and an image transformation output module. The operation efficiency is improved, and the anti-shake real-time performance is improved.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology for video, and specifically relates to a real-time video stabilization method and system based on IMU for intra-frame motion compensation. Background Technology

[0002] In the current streaming media era, more and more people are shooting short videos, vlogs, and other video footage, or using handheld action cameras for outdoor live streaming. They also need to capture and display footage in real-time on in-vehicle platforms and mobile robots. In these scenarios, the camera is often in motion, causing discontinuous footage or distortion due to continuous exposure during movement, resulting in a "rolling shutter effect." This makes it difficult for users to effectively observe and process the captured content. Therefore, video stabilization is crucial to improving the user's shooting and viewing experience. Existing video stabilization solutions include mechanical stabilization, optical stabilization, and electronic stabilization. Among them,

[0003] Mechanical image stabilization: Mechanical stabilization systems use sensors such as gyroscopes to detect camera motion and then move the entire camera image sensor in the opposite direction to compensate for the camera's motion.

[0004] Optical image stabilization: This method constructs a movable lens module system, which achieves video image stabilization by moving the lens inside the camera;

[0005] Electronic image stabilization: This method stabilizes video images by directly processing the output image. This includes feature matching to achieve motion estimation or motion measurement by sensors.

[0006] However, some existing video stabilization methods employ feature point matching, which consumes significant computational resources and is time-consuming, or require offline processing, or fail to account for the rolling shutter effect caused by rapid camera movement. Specific drawbacks include:

[0007] 1. Camera pose cannot be estimated in some low-texture scenes;

[0008] 2. Consumes computing power;

[0009] 3. A rolling shutter effect appears in the image when the camera is moving;

[0010] 4. The issue of time synchronization between the gyroscope and the image frame was not considered;

[0011] 5. When dealing with significant video shakiness, the stabilization effect was reduced too much to avoid black borders.

[0012] In addition, commonly used existing technologies include:

[0013] IMU: Inertial Measurement Unit, which can be used to measure the angular velocity of an object, and here it is used to measure the motion attitude trajectory of the camera.

[0014] Roller shutter exposure: The camera sensor scans line by line and exposes the image line by line until all pixels are exposed, thus completing the line-by-line exposure of the image.

[0015] Jelly effect: This refers to the phenomenon where the spatial pose of an image is inconsistent due to rolling exposure during camera movement.

[0016] Intra-frame stabilization: This is used to solve the intra-frame rolling shutter effect, so that the spatial pose of the image frame is uniform within the frame, and to stabilize the image between frames.

[0017] Quaternions: can be used to represent the spatial orientation of an object or the rotation of a spatial vector. Summary of the Invention

[0018] To address the aforementioned problems, the purpose of this application is to improve the user's video viewing experience, increase the computational efficiency of video stabilization, and enhance the real-time performance of video stabilization. This invention provides a real-time video stabilization method and system based on IMU intra-frame motion compensation, specifically addressing video jitter and the rolling shutter effect during motion. Furthermore,

[0019] 1. Camera pose can be measured in some low-texture scenes.

[0020] 2. Gyroscope data is used to calculate camera attitude, resulting in high image stabilization efficiency.

[0021] 3. The issue of time synchronization between gyroscope and image frame data has been addressed, reducing deviations.

[0022] 4. Solve the rolling shutter effect caused by exposure during camera movement.

[0023] 5. Layered intra-frame compensation improves computational efficiency.

[0024] 6. For videos with non-regular quadrilateral borders after image stabilization, black border detection is performed.

[0025] 7. When dealing with significant video shakiness, perform black border removal to retain more of the stabilization effect.

[0026] Specifically, the present invention provides a real-time video stabilization method based on IMU for intra-frame motion compensation, comprising the following steps:

[0027] S1, Data Preprocessing, including: data acquisition and IMU data processing;

[0028] S1.1, Data Acquisition, including:

[0029] Hardware synchronization is performed by acquiring the current frame image data Frame_data of the real-time video stream and receiving the electrical signal that the camera sensor has started to expose, and then obtaining the timestamp Frame_timestamp of the camera sensor as Frame_info.

[0030] Obtain the IMU data (IMU_data) and timestamp (imu_timestamp) from the completion of the exposure of the current frame image to the completion of the exposure of the previous frame image as IMU_info;

[0031] The camera's intrinsic parameter matrix K is obtained by calibrating the camera.

[0032] Camera calibration is used to obtain the mapping relationship between the physical reality of three-dimensional point locations and the corresponding points in the pixel coordinate system of the camera's two-dimensional image. The steps of camera calibration are as follows:

[0033] 1) Capture approximately 20 images containing the calibration plate;

[0034] 2) Detect corner points on the chessboard image;

[0035] Detecting corner positions on a checkerboard image using image processing.

[0036] 3) Calculate the intrinsic parameter matrix K;

[0037] The four coordinate systems involved in the camera imaging process are as follows:

[0038] World coordinate system: A world coordinate system that uses the real world as a reference to describe the pose of objects and cameras in the real physical world. The origin can be chosen independently. A point in the world coordinate system can be represented as (X... w Y w Z w ) T ;

[0039] Camera coordinate system: The optical center of the camera is taken as the origin, the optical axis of the camera is taken as the z-axis, and the x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system; a point in the camera coordinate system can be represented as (x c y c , z c ) T ;

[0040] Image coordinate system: The center of the camera's imaging plane is taken as the origin, and the x-axis and y-axis are parallel to two sides of the imaging plane, respectively; a point in the image coordinate system can be represented as (x, y, 1). T ;

[0041] Pixel coordinate system: The top-left corner of the imaging plane is taken as the origin, and the x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system, respectively; a point in the pixel coordinate system can be represented as (u, v, 1). T ;

[0042] The relationship between a point in the pixel coordinate system and a point in the world coordinate system is shown in the following formula:

[0043]

[0044] Since the size of each square on the captured chessboard is fixed and known, it is assumed that the coordinates of each corner point in the world coordinate system can be obtained and calculated. By using the detected corner point positions in the pixel coordinate system, and the coordinates of the corner points in the world coordinate system obtained from the chessboard size, the required intrinsic parameter matrix K can be solved by solving a series of equations based on the formulas corresponding to the aforementioned coordinate systems.

[0045]

[0046] The time frame (FrameReadDuration) is obtained based on the camera parameters to complete the exposure of the entire frame image and read the entire frame image data.

[0047] S1.2, IMU data processing, including:

[0048] By rigidly connecting the IMU sensor and the camera sensor, they can be considered as a single rigid body. Since the angular velocity is the same at all points on the rigid body, the camera's attitude change is indirectly obtained based on the IMU's raw gyrodata[x, y, z], the gyroscale coefficient gcalse calculated based on the range, and the relative position of the IMU coordinate system and the camera coordinate system. This indirect acquisition of the camera's attitude change is achieved by indirectly obtaining the camera attitude quaternion from the IMU's raw gyrodata[x, y, z], which is then obtained through the camera attitude calculation step below.

[0049] The time interval dt of the IMU data is obtained based on the sampling frequency frequency of the IMU data.

[0050] dt = 1 / frequency;

[0051] The three-axis angular velocities of the gyroscope can be obtained as follows:

[0052] ω x =gyrodata.x*gscale,

[0053] ω y =gyrodata.y*gscale,

[0054] ωz =gyrodata.z*gscale;

[0055] S2, trajectory smoothing processing, includes:

[0056] S2.1, Current camera pose calculation:

[0057] The camera attitude is calculated by integrating the camera's angular velocity based on IMU data, and the camera's rotation quaternion is obtained.

[0058] The three angular velocity components measured by a three-axis gyroscope can be combined into an angular velocity vector. This can be understood as the camera rotating around this angular velocity vector, with the angle of rotation being the integral of the magnitude of the angular velocity vector.

[0059] The camera's pose quaternion can be obtained by integrating this data, i.e.

[0060]

[0061] S2.2, Obtain the camera's true motion trajectory:

[0062] The real motion trajectory of the camera is saved by using the IMU timestamp (IMU_timestamp) and the camera attitude quaternion calculated from the IMU data (quaternion). Here, saving refers to storing the IMU timestamp (IMU_timestamp) and the camera attitude quaternion calculated from the IMU data as a set of data into a deque queue as the real motion trajectory (real_quaternion_trajectory).

[0063] S2.3, Obtain the virtual trajectory after smoothing filtering:

[0064] The quaternions of adjacent frames are subjected to spherical linear interpolation based on a smoothing coefficient α, where α∈[0,1], to obtain a smoothed quaternion V, which is then saved as a smoothed virtual quaternion trajectory virtual_quaternion_trajectory. This saving process involves storing the current frame image timestamp and pose quaternion V as a set of data into a deque queue as the virtual quaternion trajectory virtual_quaternion_trajectory.

[0065]

[0066] The spherical linear interpolation function slerp() calculates t based on the above data and formula.i The quaternion at time t, where α is calculated by the formula below.

[0067]

[0068] quaternion1 is the quaternion pose at time t1, quaternion2 is the quaternion pose at time t2, and α is the current t. i The proportion of quaternions at time points during the time interval from t1 to t2;

[0069] S3, Intra-frame motion compensation, performs intra-frame motion compensation to address the rolling shutter effect, including:

[0070] S3.1. Initialize the line exposure delay time to mLineDelay;

[0071] S3.2. Calculate the time difference timeDifference_v between the current v-th row and the middle row mHeight / 2 in the current frame image;

[0072] S3.3. Obtain the smoothed camera virtual pose (VirtualPose) and the real camera pose (RealPose) of each layer at the moment the camera's middle row begins exposure.

[0073] S3.4, obtain the motion compensation matrix MatrixCorrect that compensates the row of images to the virtual pose VirtualPose of the frame image;

[0074] S3.5, obtain the perspective transformation matrix of the image frame based on the camera intrinsic parameter matrix K and the motion compensation matrix MatrixCorrect;

[0075] S3.6, perform intra-frame motion compensation on the image layers to obtain the initial motion compensation matrix queue;

[0076] S3.7, Black border removal from the motion compensation matrix queue; obtain the motion compensation matrix queue after black border removal;

[0077] S4, Image Transformation Output, including:

[0078] The cropping magnification matrix Trimscale_Matrix of the image is calculated based on the cropping ratio, and the cropping magnification compensation transformation matrix of the image is also calculated.

[0079] MatrixFrameTransform=MatrixFrame*Trimscale_Matrix;

[0080] Based on the final motion compensation matrix queue, the image is motion compensated pixel by pixel and then cropped and enlarged to output a stabilized real-time video stream.

[0081] In step S1.2, if the sampling frequency is 500 Hz, then the time interval is 0.002 s.

[0082] Step S3 further includes:

[0083] Let v be the row in the image corresponding to point p, mHeight be the total number of rows in the image, FrameReadDuration be the time required to read the entire frame row by row from top to bottom, and Ti be the timestamp of the start of exposure of the middle row of the i-th frame. Then the intra-frame motion compensation process further includes:

[0084] S3.0. Start intra-frame motion compensation;

[0085] S3.1. Initialize the line exposure delay time to mLineDelay.

[0086] mLineDelay=FrameReadDuration / mHeight;

[0087] S3.2. Calculate the time difference timeDifference_v between the current v-th row and the middle row mHeight / 2 in the current frame image.

[0088] timeDifference_v=mLineDelay*(v-mHeight / 2);

[0089] S3.3. Obtain the smoothed pose of the camera at the moment the middle row of the camera begins exposure as the smoothed virtual pose of the current frame.

[0090] VirtualPose=virtual_quaternion_trajectory.get_interpolate(Ti);

[0091] To obtain the pose at the current time in the pose trajectory, we need to use get_interpolate(Ti) to find the virtual camera poses V1 and V2 corresponding to the two timestamps t1 and t2 adjacent to Ti in the virtual camera pose trajectory virtual_quaternion_trajectory. Then, we can calculate the scaling factor s = (Ti-t1) / (t1-t2) at the current time to obtain the virtual camera pose VirtualPose corresponding to the video frame at time Ti, VirtualPose = slerp(V1, V2, s).

[0092] Then, based on time timeDifference_v+Ti, interpolate from the real camera pose trajectory real_quaternion_trajectory to obtain the real pose RealPose starting from the v-th row of the i-th frame image and the inverse RealPoseInv, which is the pose transformation matrix that will transform the current pose to the initial pose. RealPose = real_quaternion_trajectory.get_interpolate(timeDifference_v+Ti).

[0093] S3.4. Obtain the motion compensation matrix MatrixCorrect, which compensates the image in this row to the virtual pose VirtualPose of this frame.

[0094] MatrixCorrect=RealPoseInv*VirtualPose;

[0095] S3.5. After obtaining the motion compensation matrix of the image for that row of the image frame, calculate the image frame transformation matrix MatrixFrame based on the motion compensation matrix and the camera intrinsic parameter matrix K. Compensate and correct the rolling shutter effect for each image data P(u,v) of this row of images.

[0096] MatrixFrame=K*MatrixCorrect*K_inv;

[0097] S3.6. The computation of motion compensation matrix for line-by-line compensated image data is relatively large. It can be divided into n regions according to the image height, and it is assumed that the images in the same region have the same motion vector, so as to obtain n motion compensation matrices of the image.

[0098] S3.7. The frame border of the image after intra-frame stabilization has also changed, which may cause a black border to appear in the central region of interest, so black border removal is required; here, intra-frame black border removal is performed to ensure that no meaningless black areas appear in the frame after motion compensation and to ensure that the motion compensation effect is preserved as much as possible after black border removal.

[0099] Based on the cropping ratio `trim_ratio` (trim_ratio ∈ (0.05, 0.15)), calculate `trim_width` and `trim_height`. Crop the image, preserving the central region of interest (the inner box region), then enlarge it to the original size. Remove the black borders from the stabilized image.

[0100] trim_width=mWidth*trim_ratio,

[0101] trim_height=mHeight*trim_ratio.

[0102] Step S3.7 further includes the following process for intra-frame image stabilization and black border removal:

[0103] S3.7.1, Input the initial motion compensation queue to remove black borders; that is, remove black borders from the motion compensation matrix queue of the image frame;

[0104] S3.7.2, Motion Compensation Matrix Queue Reduction: The current motion compensation matrix queue is calculated by interpolation based on the current motion compensation coefficients; that is, the motion compensation matrix queue of the image frame is calculated based on the current motion compensation coefficients and the initial motion compensation matrix queue.

[0105] S3.7.3, Image frame border calculation: Previously, the obtained image was a quadrilateral after perspective transformation of the entire frame. Now, the image border after intra-frame compensation is an irregular polygon. It is necessary to calculate the compensated image frame border based on the number of layers; that is, calculate the compensated image frame border based on the motion compensation matrix queue of the current frame image.

[0106] S3.7.4, Black Border Detection: Calculate the relationship between the compensated image frame border and the central ROI region. If the central ROI region is within the compensated image frame border, no black border appears, and the current motion compensation coefficient can be output. Conversely, if the central ROI region is not within the compensated image frame border, a black border appears, and the motion compensation coefficient needs to be reduced until a motion compensation coefficient without a black border is obtained. In other words, determine whether the calculated central ROI region is within the compensated image frame border. If not, a black border appears in the image frame motion compensation, and the motion compensation coefficient is reduced to decrease the motion compensation effect. If yes, no black border appears, and the motion compensation coefficient is output.

[0107] S3.7.5: Obtain and output the motion compensation coefficients that do not show black edges after compensation in the current frame compensation matrix queue, and calculate the final motion compensation matrix queue.

[0108] This application also relates to a real-time video stabilization system based on IMU for intra-frame motion compensation. The system is applicable to any of the above methods and includes: a data preprocessing module, a trajectory smoothing module, an intra-frame motion compensation module, and an image transformation output module; the data is processed by the data preprocessing module and then enters the trajectory smoothing module; then the intra-frame motion compensation module is used; and finally the image transformation output module is used for processing.

[0109] The system further includes:

[0110] The data preprocessing module is divided into a data acquisition module and an IMU data processing module;

[0111] The data acquisition module: acquires the current frame image data Frame_data from the real-time video stream and receives the electrical signal indicating the start of exposure of the camera sensor through hardware synchronization to obtain the timestamp Frame_timestamp of the start of exposure of the camera sensor as Frame_info; acquires the IMU data IMU_data and timestamp imu_timestamp from the completion of the current frame image exposure to the completion of the previous frame image exposure as IMU_info; calibrates the camera to obtain the camera's intrinsic parameter matrix K; the camera calibration is to obtain the mapping relationship between the three-dimensional point positions in the physical world and the corresponding points in the pixel coordinate system of the camera's two-dimensional image; the camera calibration steps are as follows:

[0112] 1) Capture approximately 20 images containing the calibration plate;

[0113] 2) Detect corner points on the chessboard image by detecting the positions of corner points in the chessboard image through image processing;

[0114] 3) Calculate the intrinsic parameter matrix K;

[0115] As described above, the required intrinsic parameter matrix K is obtained by solving the simultaneous equations based on the formulas corresponding to the coordinate system.

[0116]

[0117] The time frame (FrameReadDuration) is obtained based on the camera parameters to complete the exposure of the entire frame image and read the entire frame image data.

[0118] The IMU data processing module: The IMU sensor and camera sensor are rigidly connected, and can be considered as a single rigid body with the same angular velocity at all points. The camera's attitude change can be indirectly obtained from the raw IMU gyrodata[x, y, z], the gyroscale coefficient gcalse calculated based on the range, and the relative position of the IMU coordinate system and the camera coordinate system. The step of indirectly obtaining the camera attitude quaternion from the raw IMU gyrodata[x, y, z] is handled by the camera attitude calculation module below. The time interval dt = 1 / frequency of the IMU data can be obtained from the sampling frequency frequency. The three-axis angular velocities of the gyroscope can then be obtained.

[0119] ω x =gyrodata.x*gscale

[0120] ω y =gyrodata.y*gscale

[0121] ω z =gyrodata.z*gscale;

[0122] The trajectory smoothing module further includes:

[0123] Current camera attitude calculation: The camera attitude is calculated by integrating the camera's angular velocity based on the IMU data, and the camera's rotation quaternion is obtained.

[0124] The three angular velocity components measured by the three-axis gyroscope can be combined into an angular velocity vector. This can be understood as the camera rotating around this angular velocity vector, with the angle of rotation being the integral of the magnitude of the angular velocity vector. The camera's pose quaternion can be obtained by integrating this data.

[0125]

[0126] Camera's real motion attitude trajectory: The camera's real motion attitude trajectory, real_quaternion_trajectory, is saved based on the IMU timestamp (IMU_timestamp) and the camera's attitude quaternion calculated from the IMU data. Saving here refers to the process of storing the IMU timestamp (IMU_timestamp) and the camera's attitude quaternion calculated from the IMU data as a set of data into a deque queue as the real motion attitude trajectory, real_quaternion_trajectory.

[0127] The smoothed virtual trajectory: Based on the smoothing coefficient α (α∈[0,1]), spherical linear interpolation is performed on the quaternions of adjacent frames to obtain the smoothed quaternion V, which is then saved as the smoothed virtual quaternion trajectory virtual_quaternion_trajectory. This saving process involves storing the current frame image timestamp and pose quaternion V as a set of data into a deque queue as the virtual quaternion trajectory virtual_quaternion_trajectory.

[0128]

[0129] The spherical linear interpolation function slerp() calculates t based on the above data and formula. i The quaternion at time α; where α is calculated by the formula below.

[0130]

[0131] quaternion1 is the quaternion pose at time t1, quaternion2 is the quaternion pose at time t2, and α is the current t. i The proportion of quaternions at time points during the time interval from t1 to t2;

[0132] The intra-frame motion compensation module further includes:

[0133] Since the camera sensor uses rolling shutter exposure, the camera's spatial orientation changes during camera movement, resulting in different camera orientations for each row of pixels during exposure, which can easily cause rolling shutter effect; intra-frame motion compensation is performed to address this issue.

[0134] Let v be the row in the image corresponding to point p, mHeight be the total number of rows in the image, FrameReadDuration be the time required to read the complete frame row by row from top to bottom, and Ti be the timestamp of the middle row of the i-th frame starting to be exposed.

[0135] The image transformation output module further includes:

[0136] The cropping magnification matrix Trimscale_Matrix of the image is calculated based on the cropping ratio, and the cropping magnification compensation transformation matrix of the image is also calculated.

[0137] MatrixFrameTransform=MatrixFrame*Trimscale_Matrix;

[0138] Based on the final motion compensation matrix queue, the image is motion compensated pixel by pixel and then cropped and enlarged to output a stabilized real-time video stream.

[0139] The intra-frame motion compensation module includes the following steps:

[0140] 0. Start intra-frame motion compensation;

[0141] 1. Initialize the line exposure delay time to mLineDelay, where mLineDelay = FrameReadDuration / mHeight;

[0142] 2. Calculate the time difference timeDifference_v between the current v-th row and the middle row mHeight / 2 in the current frame image, timeDifference_v=mLineDelay*(v-mHeight / 2);

[0143] 3. Obtain the smoothed pose at the moment the camera's middle row begins exposure as the smoothed virtual pose of the current frame, VirtualPose, VirtualPose = virtual_quaternion_trajectory.get_interpolate(Ti); To obtain the pose at the current moment in the pose trajectory, use get_interpolate(Ti) to find the virtual camera poses V1 and V2 corresponding to the two timestamps t1 and t2 adjacent to Ti in the virtual camera pose trajectory virtual_quaternion_trajectory. Calculate the scaling factor s = (Ti-t1) / (t1-t2) at the current moment to obtain the virtual camera pose VirtualPose corresponding to the video frame at time Ti, VirtualPose = slerp(V1, V2, s);

[0144] Then, based on time timeDifference_v+Ti, interpolate from the real camera pose trajectory real_quaternion_trajectory to obtain the real pose RealPose starting from the v-th row of the i-th frame image and the inverse RealPoseInv, which is the pose transformation matrix that will transform the current real pose to the initial pose. RealPose = real_quaternion_trajectory.get_interpolate(timeDifference_v+Ti).

[0145] 4. Obtain the motion compensation matrix MatrixCorrect, which compensates the image in this row to the virtual pose VirtualPose of this frame image. MatrixCorrect = RealPoseInv * VirtualPose;

[0146] 5. After obtaining the motion compensation matrix of the image in this row of the image frame, calculate the image frame transformation matrix MatrixFrame based on the motion compensation matrix and the camera intrinsic parameter matrix K. Compensate and correct the rolling shutter effect for each image data P(u,v) in this row of images. MatrixFrame = K * MatrixCorrect * K_inv;

[0147] 6. The computation of motion compensation matrix for line-by-line compensated image data is relatively large. It can be divided into n regions according to the image height, and it is assumed that the images in the same region have the same motion vector, so as to obtain n motion compensation matrices of the image.

[0148] 7. After intra-frame stabilization, the frame borders of the image also change, which may cause a black border to appear in the central region of interest. Black border removal is then necessary. Intra-frame black border removal uses an intra-frame black border removal module to ensure that meaningless black areas do not appear within the image after motion compensation while preserving as much motion compensation effect as possible. This involves calculating trim_width and trim_height based on the trim ratio trim_ratio (trim_ratio∈(0.05, 0.15)), cropping to retain the central region of interest (the middle square area), then enlarging it to its original size to remove the black borders from the stabilized image.

[0149] trim_width=mWidth*trim_ratiotrim_height=mHeight*trim_ratio.

[0150] The intra-frame black border removal module further includes:

[0151] Input the initial motion compensation queue to remove black borders;

[0152] Motion compensation matrix queue reduction: The current motion compensation matrix queue is calculated by interpolation based on the current motion compensation coefficients;

[0153] Image frame border calculation: Previously, the obtained image was a quadrilateral after perspective transformation of the entire frame. Now, the image border after intra-frame compensation is an irregular polygon. The compensated image frame border needs to be calculated based on the number of layers.

[0154] Black border detection: Calculate the relationship between the compensated image frame border and the central ROI region. If the central ROI region is within the compensated image frame border, no black border appears, and the current motion compensation coefficient can be output. Conversely, if the central ROI region is not within the compensated image frame border, a black border appears, and the motion compensation coefficient needs to be reduced until a motion compensation coefficient without black borders is obtained.

[0155] Obtain and output the motion compensation coefficients that do not show black edges after compensation in the current frame compensation matrix queue, and calculate the final motion compensation matrix queue.

[0156] Therefore, the advantage of this application is:

[0157] 1. Measuring camera attitude by collecting IMU data requires relatively little computing power and can perform real-time video stabilization on real-time video streams.

[0158] 2. Using IMU data, camera pose can be measured to achieve video stabilization even when image features are not obvious due to motion blur, lack of texture in the shooting scene, and weak light intensity.

[0159] 3. Hardware time synchronization between the frame and the IMU is achieved by acquiring the electrical signal at which the camera sensor begins exposure.

[0160] 4. Using IMU data for intra-frame image stabilization can compensate for the rolling shutter effect caused by camera movement.

[0161] 5. Use IMU data to perform layered motion compensation on image frame data to improve computational efficiency for intra-frame stabilization.

[0162] 6. For videos with non-regular quadrilateral borders after image stabilization, black border detection is performed.

[0163] 7. In-frame stabilization removes black borders, which can avoid black borders in the image while preserving the stabilization effect as much as possible. Attached Figure Description

[0164] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0165] Figure 1 This is a flowchart of the video stabilization method with intra-frame motion compensation in this application.

[0166] Figure 2 This is a schematic diagram of the transformation from the world coordinate system to the pixel coordinate system in this application.

[0167] Figure 3 This is a schematic diagram of the corner points on the checkerboard image detected during camera calibration in this application.

[0168] Figure 4 This is a schematic diagram of a point image from the physical real world being projected onto a camera image in this application.

[0169] Figure 5 This is a schematic diagram of intra-frame motion compensation for the jelly effect in this application.

[0170] Figure 6 This is a flowchart of the intra-frame motion compensation process in this application.

[0171] Figure 7 This is a schematic diagram showing the black borders at the edges of the image frame and the central region of interest after intra-frame stabilization in this application.

[0172] Figure 8 This is a schematic diagram of image cropping, enlarging, and removing black borders in this application.

[0173] Figure 9 This is a flowchart of the intra-frame stabilization and black border removal process in this application.

[0174] Figure 10 This is a schematic block diagram of the video stabilization system with intra-frame motion compensation in this application. Detailed Implementation

[0175] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0176] This invention relates to a real-time video stabilization method and system based on IMU data for intra-frame motion compensation. This solution is applied in the field of video image stabilization, such as... Figure 1 As shown, the method includes the following steps:

[0177] S1, Data Preprocessing, including: data acquisition and IMU data processing;

[0178] S1.1, Data Acquisition, including:

[0179] Hardware synchronization is performed by acquiring the current frame image data Frame_data of the real-time video stream and receiving the electrical signal that the camera sensor has started to expose, and then obtaining the timestamp Frame_timestamp of the camera sensor as Frame_info.

[0180] Obtain the IMU data (IMU_data) and timestamp (imu_timestamp) from the completion of the exposure of the current frame image to the completion of the exposure of the previous frame image as IMU_info;

[0181] The camera's intrinsic parameter matrix K is obtained by calibrating the camera.

[0182] Camera calibration is used to obtain the mapping relationship between the physical reality of three-dimensional point locations and the corresponding points in the pixel coordinate system of the camera's two-dimensional image. The steps of camera calibration are as follows:

[0183] 1) Capture approximately 20 images containing the calibration plate; for example, you can select 18-23 images containing the calibration plate.

[0184] 2) Detect corner points on the chessboard image;

[0185] Detecting corner positions on a checkerboard image using image processing, such as... Figure 2 As shown.

[0186] 4) Calculate the intrinsic parameter matrix K.

[0187] The four coordinate systems involved in the camera imaging process are as follows: Figure 3 As shown:

[0188] World coordinate system: A world coordinate system that uses the real world as a reference to describe the pose of objects and cameras in the real physical world. The origin can be chosen independently. A point in the world coordinate system can be represented as (X... w Y w Zw ) T .

[0189] Camera coordinate system: The camera's optical center is used as the origin, the camera's optical axis is used as the z-axis, and the x and y axes are parallel to the x and y axes of the image coordinate system. A point in the camera coordinate system can be represented as (x... c y c , z c ) T .

[0190] Image coordinate system: The center of the camera's imaging plane is taken as the origin, and the x-axis and y-axis are parallel to two sides of the imaging plane, respectively. A point in the image coordinate system can be represented as (x, y, 1). T .

[0191] Pixel coordinate system: The origin is set at the top-left corner of the imaging plane, and the x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system, respectively. A point in the pixel coordinate system can be represented as (u, v, 1). T .

[0192] A schematic diagram of how a camera images points from the physical world onto an image is shown below. Figure 4 As shown.

[0193] The relationship between a point in the pixel coordinate system and a point in the world coordinate system is shown in the following formula.

[0194]

[0195] Since the size of each square on the captured chessboard is fixed and known, it is assumed that the coordinates of each corner point in the world coordinate system can be obtained and calculated. By using the detected corner point positions in the pixel coordinate system, and by obtaining the calculated corner point coordinates in the world coordinate system from the chessboard size, the required intrinsic parameter matrix K can be solved by solving a series of equations based on the formulas corresponding to the aforementioned coordinate systems.

[0196]

[0197] The time frame (FrameReadDuration) is obtained based on the camera parameters to complete the exposure of the entire frame image and read the entire frame image data.

[0198] S1.2, IMU data processing, including:

[0199] By rigidly connecting the IMU sensor and the camera sensor, they can be considered as a single rigid body. Since the angular velocity is the same at all points on the rigid body, the camera's attitude change is indirectly obtained based on the IMU's raw gyrodata[x, y, z], the gyroscale coefficient gcalse calculated based on the range, and the relative position of the IMU coordinate system and the camera coordinate system. This indirect acquisition of the camera's attitude change is achieved by indirectly obtaining the camera attitude quaternion from the IMU's raw gyrodata[x, y, z], which is then obtained through the camera attitude calculation step below.

[0200] The time interval dt of the IMU data is obtained based on the sampling frequency frequency of the IMU data.

[0201] dt = 1 / frequency;

[0202] The three-axis angular velocities of the gyroscope can be obtained as follows:

[0203] ω x =gyrodata.x*gscale,

[0204] ω y =gyrodata.y*gscale,

[0205] ω z =gyrodata.z*gscale;

[0206] S2, trajectory smoothing processing, includes:

[0207] S2.1, Current camera pose calculation:

[0208] This step is the camera attitude calculation mentioned above. Based on the IMU data, the camera's angular velocity is integrated to calculate the camera attitude and obtain the camera's rotation quaternion.

[0209] The three angular velocity components measured by a three-axis gyroscope can be combined into an angular velocity vector. This can be understood as the camera rotating around this angular velocity vector, with the angle of rotation being the integral of the magnitude of the angular velocity vector.

[0210] The camera's pose quaternion can be obtained by integrating this data, i.e.

[0211]

[0212] S2.2, Obtain the camera's true motion trajectory:

[0213] The real motion trajectory of the camera is saved by using the IMU timestamp (IMU_timestamp) and the camera attitude quaternion calculated from the IMU data (quaternion). Here, saving refers to storing the IMU timestamp (IMU_timestamp) and the camera attitude quaternion calculated from the IMU data as a set of data into a deque queue as the real motion trajectory (real_quaternion_trajectory).

[0214] S2.3, Obtain the virtual trajectory after smoothing filtering:

[0215] The quaternions of adjacent frames are subjected to spherical linear interpolation based on a smoothing coefficient α, where α∈[0,1], to obtain a smoothed quaternion V, which is then saved as a smoothed virtual quaternion trajectory virtual_quaternion_trajectory. This saving process involves storing the current frame image timestamp and pose quaternion V as a set of data into a deque queue as the virtual quaternion trajectory virtual_quaternion_trajectory.

[0216]

[0217] The spherical linear interpolation function slerp() calculates t based on the above data and formula. i The quaternion at time t, where α is calculated by the formula below.

[0218]

[0219] quaternion1 is the quaternion pose at time t1, quaternion2 is the quaternion pose at time t2, and α is the current t. i The proportion of quaternions at time points during the time interval from t1 to t2;

[0220] S3, Intra-frame motion compensation, performs intra-frame motion compensation to address the rolling shutter effect, including:

[0221] Let v be the row in the image corresponding to point p, mHeight be the total number of rows in the image, FrameReadDuration be the time required to read the entire frame row by row from top to bottom, and Ti be the timestamp of the start of exposure of the middle row of the i-th frame. Then the intra-frame motion compensation process further includes:

[0222] S3.0. Start intra-frame motion compensation;

[0223] S3.1. Initialize the line exposure delay time to mLineDelay.

[0224] mLineDelay=FrameReadDuration / mHeight;

[0225] S3.2. Calculate the time difference timeDifference_v between the current v-th row and the middle row mHeight / 2 in the current frame image.

[0226] timeDifference_v=mLineDelay*(v-mHeight / 2);

[0227] S3.3. Obtain the smoothed pose of the camera at the moment the middle row of the camera begins exposure as the smoothed virtual pose of the current frame.

[0228] VirtualPose=virtual_quaternion_trajectory.get_interpolate(Ti);

[0229] To obtain the pose at the current time in the pose trajectory, we need to use get_interpolate(Ti) to find the virtual camera poses V1 and V2 corresponding to the two timestamps t1 and t2 adjacent to Ti in the virtual camera pose trajectory virtual_quaternion_trajectory. Then, we can calculate the scaling factor s = (Ti-t1) / (t1-t2) at the current time to obtain the virtual camera pose VirtualPose corresponding to the video frame at time Ti, VirtualPose = slerp(V1, V2, s).

[0230] Then, based on timeDifference_v+Ti, interpolate from the real camera pose trajectory real_quaternion_trajectory to obtain the real pose RealPose starting from row v in the i-th frame image and the inverse RealPoseInv, which is the pose transformation matrix that transforms the current pose to the initial pose.

[0231] RealPose=real_quaternion_trajectory.get_interpolate(timeDifference_v+Ti);

[0232] S3.4. Obtain the motion compensation matrix MatrixCorrect, which compensates the image in this row to the virtual pose VirtualPose of this frame.

[0233] MatrixCorrect=RealPoseInv*VirtualPose;

[0234] S3.5. After obtaining the motion compensation matrix of the image for that row of the image frame, calculate the image frame transformation matrix MatrixFrame based on the motion compensation matrix and the camera intrinsic parameter matrix K. Compensate and correct the rolling shutter effect for each image data P(u, V) of this row of images.

[0235] MatrixFrame=K*MatrixCorrect*K_inv;

[0236] S3.6. The computation of motion compensation matrix for line-by-line compensated image data is relatively large. It can be divided into n regions according to the image height, and it is assumed that the images in the same region have the same motion vector, so as to obtain n motion compensation matrices of the image.

[0237] S3.7. The frame border of the image after intra-frame stabilization has also changed, which may cause a black border to appear in the central region of interest, so black border removal is required; here, intra-frame black border removal is performed to ensure that no meaningless black areas appear in the frame after motion compensation and to ensure that the motion compensation effect is preserved as much as possible after black border removal.

[0238] Based on the cropping ratio `trim_ratio` (trim_ratio ∈ (0.05, 0.15)), calculate `trim_width` and `trim_height`. Crop the image, preserving the central region of interest (the inner box region), then enlarge it to the original size. Remove the black borders from the stabilized image.

[0239] trim_width=mWidth*trim_ratio,

[0240] trim_height=mHeight*trim_ratio;

[0241] Step S3.7 further includes: the intra-frame image stabilization and black border removal process as follows: Figure 9 As shown:

[0242] S3.7.1, Input the initial motion compensation queue to remove black borders; that is, remove black borders from the motion compensation matrix queue of the image frame;

[0243] S3.7.2, Motion Compensation Matrix Queue Reduction: The current motion compensation matrix queue is calculated by interpolation based on the current motion compensation coefficients; that is, the motion compensation matrix queue of the image frame is calculated based on the current motion compensation coefficients and the initial motion compensation matrix queue.

[0244] S3.7.3, Image frame border calculation: Previously, the obtained image was a quadrilateral after perspective transformation of the entire frame. Now, the image border after intra-frame compensation is an irregular polygon. It is necessary to calculate the compensated image frame border based on the number of layers; that is, calculate the compensated image frame border based on the motion compensation matrix queue of the current frame image.

[0245] S3.7.4, Black Border Detection: Calculate the relationship between the compensated image frame border and the central ROI region. If the central ROI region is within the compensated image frame border, no black border appears, and the current motion compensation coefficient can be output. Conversely, if the central ROI region is not within the compensated image frame border, a black border appears, and the motion compensation coefficient needs to be reduced until a motion compensation coefficient without a black border is obtained. In other words, determine whether the calculated central ROI region is within the compensated image frame border. If not, a black border appears in the image frame motion compensation, and the motion compensation coefficient is reduced to decrease the motion compensation effect. If yes, no black border appears, and the motion compensation coefficient is output.

[0246] S3.7.5, obtain and output the motion compensation coefficients that do not show black edges after compensation in the current frame compensation matrix queue, and calculate the final motion compensation matrix queue;

[0247] S4, Image Transformation Output, including:

[0248] The cropping magnification matrix Trimscale_Matrix of the image is calculated based on the cropping ratio, and the cropping magnification compensation transformation matrix of the image is also calculated.

[0249] MatrixFrameTransform=MatrixFrame*Trimscale_Matrix;

[0250] Based on the final motion compensation matrix queue, the image is motion compensated pixel by pixel and then cropped and enlarged to output a stabilized real-time video stream.

[0251] The system involved in this application is applicable to the above method, and its implementation is as follows: Figure 10 As shown, it includes: a data preprocessing module, a trajectory smoothing module, an intra-frame motion compensation module, and an image transformation output module. The system proceeds from the data preprocessing module to the trajectory smoothing module; then the intra-frame motion compensation module; and finally the image transformation output module.

[0252] The data preprocessing module is mainly divided into a data acquisition module and an IMU data processing module.

[0253] The data acquisition module acquires the current frame image data (Frame_data) from the real-time video stream and receives the electrical signal indicating the start of exposure of the camera sensor. It then uses hardware synchronization to obtain the timestamp (Frame_timestamp) of the camera sensor's exposure start as Frame_info. Finally, it acquires the IMU data (IMU_data) and timestamp (imu_timestamp) from the completion of the current frame's exposure to the completion of the previous frame's exposure, also as IMU_info. The module calibrates the camera to obtain its intrinsic parameter matrix K. Camera calibration aims to obtain the mapping relationship between the positions of three-dimensional points in the real physical world and the corresponding points in the pixel coordinate system of the camera's two-dimensional image. The camera calibration steps are as follows.

[0254] 1) Take and collect approximately 20 images containing the calibration plate.

[0255] 2) Detect corner points on the chessboard image.

[0256] Detecting corner positions on a checkerboard image using image processing, such as... Figure 2 As shown.

[0257] 3) Calculate the intrinsic parameter matrix K.

[0258] As mentioned above, the camera imaging process involves four coordinate systems as follows: Figure 3 As shown:

[0259] The relationship between a point in the pixel coordinate system and a point in the world coordinate system is shown in the following formula:

[0260]

[0261] The required intrinsic parameter matrix K can be obtained by solving the simultaneous equations based on the formulas corresponding to the above coordinate system.

[0262]

[0263] The time frame (FrameReadDuration) is obtained based on the camera parameters to complete the exposure of the entire frame image and read the entire frame image data.

[0264] The IMU data processing module rigidly connects the IMU sensor and the camera sensor, treating them as a single rigid body with uniform angular velocities. Camera attitude changes can be indirectly obtained from the raw IMU gyrodata[x, y, z], the gyroscale coefficient gcalse calculated based on the range, and the relative positions of the IMU and camera coordinate systems. The step of indirectly obtaining the camera attitude quaternion from the raw IMU gyrodata[x, y, z] is handled by the camera attitude calculation module below. The IMU data time interval dt = 1 / frequency can be obtained from the IMU data sampling frequency (frequency). For example, if the sampling frequency is 500 Hz, the time interval is 0.002 s. The three-axis angular velocities of the gyroscope can then be obtained.

[0265] ω x =gyrodata.x*gscale

[0266] ω y =gyrodata.y*gscale

[0267] ω z =gyrodata.z*gscale;

[0268] The trajectory smoothing module further includes:

[0269] Current camera attitude calculation: The camera attitude is calculated by integrating the camera's angular velocity based on the IMU data, and the camera's rotation quaternion is obtained.

[0270] The three angular velocity components measured by the three-axis gyroscope can be combined into an angular velocity vector. This can be understood as the camera rotating around this angular velocity vector, with the angle of rotation being the integral of the magnitude of the angular velocity vector. The camera's pose quaternion can be obtained by integrating this data.

[0271]

[0272] The camera's true motion attitude trajectory (real_quaternion_trajectory) is saved based on the IMU timestamp (IMU_timestamp) and the camera's attitude quaternion calculated from the IMU data. Saving refers to the process of storing the IMU timestamp and the calculated camera attitude quaternion as a set of data into a deque as the real motion attitude trajectory (real_quaternion_trajectory).

[0273] The smoothed virtual trajectory: Based on the smoothing coefficient α (α∈[0,1]), spherical linear interpolation is performed on the quaternions of adjacent frames to obtain the smoothed quaternion V, which is then saved as the smoothed virtual quaternion trajectory virtual_quaternion_trajectory. Saving is the process of storing the current frame image timestamp and pose quaternion V as a set of data into a deque queue as the virtual quaternion trajectory virtual_quaternion_trajectory.

[0274]

[0275] quaternion1 is the quaternion pose at time t1, quaternion2 is the quaternion pose at time t2, and α is the current t. i The proportion of quaternions at time points t1 to t2 during the time interval t1 to t2. The spherical linear interpolation function slerp() calculates the interpolation value for t based on the above data and formula. i The quaternion at time t. Where α is calculated by the formula below,

[0276]

[0277] The intra-frame motion compensation module further includes:

[0278] Because the camera sensor uses rolling shutter exposure, the camera's spatial orientation changes during camera movement, resulting in different camera positions for each row of pixels during exposure. This can easily lead to issues such as... Figure 5 The jelly effect shown.

[0279] This solution can perform intra-frame motion compensation for this situation. Let v be the row in the image corresponding to point p, mHeight be the total number of rows in the image, and FrameReadDuration be the time required to read the entire frame row by row from top to bottom. Ti is the timestamp of the start of exposure for the middle row of the i-th frame. The intra-frame motion compensation process is as follows: Figure 6 As shown.

[0280] 0. Start intra-frame motion compensation;

[0281] 1. Initialize the line exposure delay time to mLineDelay.

[0282] mLineDelay=FrameReadDuration / mHeight;

[0283] 2. Calculate the time difference (timeDifference_v) between the current v-th row and the middle row (mHeight / 2) in the current frame image.

[0284] timeDifference_v=mLineDelay*(v-mHeight / 2);

[0285] 3. Obtain the smoothed pose of the camera at the moment the middle line of the exposure begins, as the smoothed virtual pose of the current frame.

[0286] VirtualPose=virtual_quaternion_trajectory.get_interpolate(Ti);

[0287] To obtain the pose at the current time in the pose trajectory, we need to use get_interpolate(Ti) to find the virtual camera poses V1 and V2 corresponding to the two timestamps t1 and t2 adjacent to Ti in the virtual camera pose trajectory virtual_quaternion_trajectory. Then, we can calculate the scaling factor s = (Ti-t1) / (t1-t2) at the current time to obtain the virtual camera pose VirtualPose corresponding to the video frame at time Ti, VirtualPose = slerp(V1, V2, s).

[0288] Then, based on timeDifference_v+Ti, interpolate from the real camera pose trajectory real_quaternion_trajectory to obtain the real pose RealPose starting from row v in the i-th frame image and the inverse RealPoseInv, which is the pose transformation matrix that transforms the current pose to the initial pose.

[0289] RealPose=real_quaternion_trajectory.get_interpolate(timeDifference_v+Ti);

[0290] 4. Obtain the motion compensation matrix MatrixCorrect, which compensates the image in that row to the virtual pose VirtualPose of that frame.

[0291] MatrixCorrect=RealPoseInv*VirtualPose;

[0292] 5. After obtaining the motion compensation matrix for that row of images in the image frame, calculate the image frame transformation matrix MatrixFrame based on the motion compensation matrix and the camera intrinsic parameter matrix K. Then, compensate for and correct the rolling shutter effect on each image data P(u, v) of that row of images.

[0293] MatrixFrame=K*MatrixCorrect*K_inv;

[0294] 6. The computation of motion compensation matrix for line-by-line compensated image data is relatively large. It can be divided into n regions according to the image height, and it is assumed that the images in the same region have the same motion vector, so as to obtain n motion compensation matrices of the image.

[0295] 7. For example Figure 7 As shown, the frame borders of the image also change after intra-frame stabilization, which may cause a black border to appear in the central region of interest, requiring black border removal. Here, intra-frame black border removal is performed using an intra-frame black border removal module to ensure that meaningless black areas do not appear within the image frame after motion compensation, and to preserve the motion compensation effect as much as possible after black border removal.

[0296] like Figure 8 As shown, here we need to calculate trim_width and trim_height based on the trim ratio trim_ratio (trim_ratio∈(0.05,0.15)), trim the central region of interest (the inner box region) of the image, and then enlarge it to the original size to remove the black borders in the stabilized image.

[0297] trim_width=mWidth*trim_ratio

[0298] trim_height=mHeight*trim_ratio;

[0299] The workflow of the intra-frame black border removal module further includes: Figure 9 As shown:

[0300] Input the initial motion compensation queue to remove black borders;

[0301] Motion compensation matrix queue reduction: The current motion compensation matrix queue is calculated by interpolation based on the current motion compensation coefficient.

[0302] Image frame border calculation: Previously, the obtained image was a quadrilateral after perspective transformation of the entire frame. Now, the image border after intra-frame compensation is an irregular polygon. The compensated image frame border needs to be calculated based on the number of layers.

[0303] Black border detection: Calculate the relationship between the compensated image frame border and the central ROI region. If the central ROI region is within the compensated image frame border, no black border appears, and the current motion compensation coefficient can be output. Conversely, if the central ROI region is not within the compensated image frame border, a black border appears, and the motion compensation coefficient needs to be reduced until a motion compensation coefficient without black borders is obtained.

[0304] Obtain and output the motion compensation coefficients that do not show black edges after compensation in the current frame compensation matrix queue, and calculate the final motion compensation matrix queue.

[0305] The image transformation output module further includes:

[0306] The cropping magnification matrix Trimscale_Matrix of the image is calculated based on the cropping ratio, and the cropping magnification compensation transformation matrix of the image is also calculated.

[0307] MatrixFrameTransform=MatrixFrame*Trimscale_Matrix;

[0308] Based on the final motion compensation matrix queue, the image is motion compensated pixel by pixel and then cropped and enlarged to output a stabilized real-time video stream.

[0309] In summary, this application is based on:

[0310] 1. Real-time image stabilization of IMU data.

[0311] 2. Obtain the electrical signal at the start of exposure of the camera sensor to perform hardware time synchronization of the frame and IMU, and then obtain the pose data after time synchronization of the frame and IMU through spherical linear interpolation.

[0312] 3. Use IMU data to perform intra-frame image stabilization on image frame data to solve the rolling shutter effect caused by exposure during camera movement.

[0313] 4. Use IMU data to perform layered motion compensation on image frame data to improve computational efficiency for intra-frame stabilization.

[0314] 5. For videos with non-regular quadrilateral borders after image stabilization, black border detection is performed.

[0315] 6. In-frame stabilization and black border removal: While ensuring the removal of black borders, retain the stabilization effect as much as possible.

[0316] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time video stabilization method based on IMU for intra-frame motion compensation, characterized in that, The method includes the following steps: S1, Data Preprocessing, including: data acquisition and IMU data processing; S1.1, Data Acquisition, including: Hardware synchronization is performed by acquiring the current frame image data Frame_data of the real-time video stream and receiving the electrical signal that the camera sensor has started to expose, and then obtaining the timestamp Frame_timestamp of the camera sensor as Frame_info. Obtain the IMU data (IMU_data) and timestamp (imu_timestamp) from the completion of the exposure of the current frame image to the completion of the exposure of the previous frame image as IMU_info; The camera's intrinsic parameter matrix K is obtained by calibrating the camera. Camera calibration is used to obtain the mapping relationship between the physical reality of three-dimensional point locations and the corresponding points in the pixel coordinate system of the camera's two-dimensional image. The steps of camera calibration are as follows: 1) Capture approximately 20 images containing the calibration plate; 2) Detect corner points on the chessboard image; Detecting corner positions on a checkerboard image using image processing. 3) Calculate the intrinsic parameter matrix K; The four coordinate systems involved in the camera imaging process are as follows: World coordinate system: A world coordinate system that uses the real world as a reference to describe the pose of objects and cameras in the real physical world. The origin can be chosen independently. A point in the world coordinate system can be represented as (x... w ,Y w Z w ) T ; Camera coordinate system: The optical center of the camera is taken as the origin, the optical axis of the camera is taken as the z-axis, and the x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system; a point in the camera coordinate system can be represented as (x c y c , z c ) T ; Image coordinate system: The center of the camera's imaging plane is taken as the origin, and the x-axis and y-axis are parallel to two sides of the imaging plane, respectively; a point in the image coordinate system can be represented as (x, y, 1). T ; Pixel coordinate system: The top-left corner of the imaging plane is taken as the origin, and the x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system, respectively; a point in the pixel coordinate system can be represented as (u, v, 1). T ; The relationship between a point in the pixel coordinate system and a point in the world coordinate system is shown in the following formula: Since the size of each square on the captured chessboard is fixed and known, it is assumed that the coordinates of each corner point in the world coordinate system can be obtained and calculated. By using the detected corner point positions in the pixel coordinate system, and the coordinates of the corner points in the world coordinate system obtained from the chessboard size, the required intrinsic parameter matrix K can be solved by solving a series of equations based on the formulas corresponding to the aforementioned coordinate systems. The time frame (FrameReadDuration) is obtained based on the camera parameters to complete the exposure of the entire frame image and read the entire frame image data. S1.2, IMU data processing, including: By rigidly connecting the IMU sensor and the camera sensor, they can be considered as a single rigid body. Since the angular velocity is the same at all points on the rigid body, the camera's attitude change is indirectly obtained based on the raw gyrodata[x,y,z] of the IMU, the gyroscale coefficient gcalse calculated based on the range, and the relative position of the IMU coordinate system and the camera coordinate system. The indirect acquisition of the camera's attitude change is obtained by indirectly obtaining the camera attitude quaternion from the raw gyrodata[x,y,z] of the IMU, which is then obtained by the camera attitude calculation below. The time interval dt of the IMU data is obtained based on the sampling frequency frequency of the IMU data, where dt = 1 / frequency; The three-axis angular velocities of the gyroscope can be obtained as follows: ω x =gyrodata.x*gscale, w y =gyrodata.y*gscale, ω z =gyrodata.z*gscale; S2, trajectory smoothing processing, includes: S2.1, Current camera pose calculation: The camera attitude is calculated by integrating the camera's angular velocity based on IMU data, and the camera's rotation quaternion is obtained. The three angular velocity components measured by a three-axis gyroscope can be combined into an angular velocity vector. This can be understood as the camera rotating around this angular velocity vector, with the angle of rotation being the integral of the magnitude of the angular velocity vector. The camera's pose quaternion can be obtained by integrating this data, i.e. S2.2, Obtain the camera's true motion trajectory: The real motion trajectory of the camera is saved by using the IMU timestamp (IMU_timestamp) and the camera attitude quaternion calculated from the IMU data (quaternion). Here, saving refers to storing the IMU timestamp (IMU_timestamp) and the camera attitude quaternion calculated from the IMU data as a set of data into a deque queue as the real motion trajectory (real_quaternion_trajectory). S2.3, Obtain the virtual trajectory after smoothing filtering: The quaternions of adjacent frames are subjected to spherical linear interpolation based on a smoothing coefficient α, where α∈[0,1], to obtain a smoothed quaternion V, which is then saved as a smoothed virtual quaternion trajectory virtual_quaternion_trajectory. This saving process involves storing the current frame image timestamp and pose quaternion V as a set of data into a deque queue as the virtual quaternion trajectory virtual_quaternion_trajectory. The spherical linear interpolation function slerp() calculates t based on the above data and formula. i The quaternion at time t, where α is calculated by the formula below. quaternion1 is the quaternion pose at time t1, quaternion2 is the quaternion pose at time t2, and α is the current t. i The proportion of quaternions at time points during the time interval from t1 to t2; S3, Intra-frame motion compensation, performs intra-frame motion compensation to address the rolling shutter effect, including: S3.

1. Initialize the line exposure delay time to mLineDelay; S3.

2. Calculate the time difference timeDifference_v between the current v-th row and the middle row mHeight / 2 in the current frame image; S3.

3. Obtain the smoothed camera virtual pose (VirtualPose) and the real camera pose (RealPose) of each layer at the moment the camera's middle row begins exposure. S3.4, obtain the motion compensation matrix MatrixCorrect that compensates the row of images to the virtual pose VirtualPose of the frame image; S3.5, obtain the perspective transformation matrix of the image frame based on the camera intrinsic parameter matrix K and the motion compensation matrix MatrixCorrect; S3.6, perform intra-frame motion compensation on the image layers to obtain the initial motion compensation matrix queue; S3.7, Motion compensation matrix queue black border removal processing; obtain the motion compensation matrix queue after black border removal; S4, Image transformation output, including: The cropping magnification matrix Trimscale_Matrix of the image is calculated based on the cropping ratio, and the cropping magnification compensation transformation matrix of the image is also calculated. MatrixFrameTransform=MatrixFrame*Trimscale_Matrix; Based on the final motion compensation matrix queue, the image is motion compensated pixel by pixel and then cropped and enlarged to output a stabilized real-time video stream.

2. The real-time video stabilization method based on IMU for intra-frame motion compensation according to claim 1, characterized in that, In step S1.2, if the sampling frequency is 500 Hz, then the time interval is 0.002 s.

3. The real-time video stabilization method based on IMU for intra-frame motion compensation according to claim 1, characterized in that, Step S3 further includes: Let v be the row in the image corresponding to point p, mHeight be the total number of rows in the image, FrameReadDuration be the time required to read the entire frame row by row from top to bottom, and Ti be the timestamp of the start of exposure of the middle row of the i-th frame. Then the intra-frame motion compensation process further includes: S3.

0. Start intra-frame motion compensation; S3.

1. Initialize the line exposure delay time to mLineDelay. mLineDelay=FrameReadDuration / mHeight; S3.

2. Calculate the time difference timeDifference_v between the current v-th row and the middle row mHeight / 2 in the current frame image. timeDifference_v=mLineDelay*(v-mHeight / 2); S3.

3. Obtain the smoothed pose of the camera at the moment the middle row of the camera begins exposure as the smoothed virtual pose of the current frame. VirtualPose=virtual_quaternion_trajectory.get_interpolate(Ti); To obtain the pose at the current time in the pose trajectory, we need to use get_interpolate(Ti) to find the virtual camera poses V1 and V2 corresponding to the two timestamps t1 and t2 adjacent to Ti in the virtual camera pose trajectory virtual_quaternion_trajectory. Then, we can calculate the scaling factor s = (Ti-t1) / (t1-t2) at the current time to obtain the virtual camera pose VirtualPose corresponding to the video frame at time Ti, VirtualPose = slerp(V1, V2, s). Then, based on time timeDifference_v+Ti, interpolate from the real camera pose trajectory real_quaternion_trajectory to obtain the real pose RealPose starting from the v-th row of the i-th frame image and the inverse RealPoselnv of the real pose, which is the pose transformation matrix that will transform the current pose to the initial pose. RealPose = real_quaternion_trajectory.get_interpolate(timeDifference_v+Ti). S3.

4. Obtain the motion compensation matrix MatrixCorrect, which compensates the image in this row to the virtual pose VirtualPose of this frame. MatrixCorrect=RealPoselnv*VirtualPose; S3.

5. After obtaining the motion compensation matrix of the image for that row of the image frame, calculate the image frame transformation matrix MatrixFrame based on the motion compensation matrix and the camera intrinsic parameter matrix K. Compensate and correct the rolling shutter effect for each image data P(u,v) of this row of images. MatrixFrame=K*MatrixCorrect*K_inv; S3.

6. The computation of motion compensation matrix for line-by-line compensated image data is relatively large. It can be divided into n regions according to the image height, and it is assumed that the images in the same region have the same motion vector, so as to obtain n motion compensation matrices of the image. S3.

7. The frame border of the image after intra-frame stabilization has also changed, which may cause a black border to appear in the central region of interest, so black border removal is required; here, intra-frame black border removal is performed to ensure that no meaningless black areas appear in the frame after motion compensation and to ensure that the motion compensation effect is preserved as much as possible after black border removal. Based on the cropping ratio `trim_ratio` (trim_ratio ∈ (0.05, 0.15)), calculate `trim_width` and `trim_height`. Crop the image, preserving the central region of interest (the inner square region), then enlarge it to its original size. Remove the black borders from the stabilized image. trim_width=mWidth*trim_ratio, trim_height=mHeight*trim_ratio.

4. The real-time video stabilization method based on IMU for intra-frame motion compensation according to claim 3, characterized in that, Step S3.7 further includes the following process for intra-frame image stabilization and black border removal: S3.7.1, Input the initial motion compensation queue to remove black borders; that is, remove black borders from the motion compensation matrix queue of the image frame; S3.7.2, Motion Compensation Matrix Queue Reduction: The current motion compensation matrix queue is calculated by interpolation based on the current motion compensation coefficients; that is, the motion compensation matrix queue of the image frame is calculated based on the current motion compensation coefficients and the initial motion compensation matrix queue. S3.7.3, Image frame border calculation: Previously, the obtained image was a quadrilateral after perspective transformation of the entire frame. Now, the image border after intra-frame compensation is an irregular polygon. It is necessary to calculate the compensated image frame border based on the number of layers; that is, calculate the compensated image frame border based on the motion compensation matrix queue of the current frame image. S3.7.4, Black Border Detection: Calculate the relationship between the compensated image frame border and the central ROI region. If the central ROI region is within the compensated image frame border, no black border appears, and the current motion compensation coefficient can be output. Conversely, if the central ROI region is not within the compensated image frame border, a black border appears, and the motion compensation coefficient needs to be reduced until a motion compensation coefficient without a black border is obtained. In other words, determine whether the calculated central ROI region is within the compensated image frame border. If not, a black border appears in the image frame motion compensation, and the motion compensation coefficient is reduced to decrease the motion compensation effect. If yes, no black border appears, and the motion compensation coefficient is output. S3.7.5: Obtain and output the motion compensation coefficients that do not show black edges after compensation in the current frame compensation matrix queue, and calculate the final motion compensation matrix queue.

5. A real-time video stabilization system based on IMU for intra-frame motion compensation, characterized in that, The system is applicable to any one of the methods of claims 1-4 above, including: a data preprocessing module, a trajectory smoothing module, an intra-frame motion compensation module, and an image transformation output module; after processing by the data preprocessing module, the data enters the trajectory smoothing module; then the intra-frame motion compensation module is used; and finally the image transformation output module is used for processing.

6. A real-time video stabilization system based on IMU for intra-frame motion compensation according to claim 5, characterized in that, The system further includes: The data preprocessing module is divided into a data acquisition module and an IMU data processing module; The data acquisition module: acquires the current frame image data Frame_data from the real-time video stream and receives the electrical signal indicating the start of exposure of the camera sensor through hardware synchronization to obtain the timestamp Frame_timestamp of the start of exposure of the camera sensor as Frame_info; acquires the IMU data IMU_data and timestamp imu_timestamp from the completion of the current frame image exposure to the completion of the previous frame image exposure as IMU_info; calibrates the camera to obtain the camera's intrinsic parameter matrix K; the camera calibration is to obtain the mapping relationship between the three-dimensional point positions in the physical world and the corresponding points in the pixel coordinate system of the camera's two-dimensional image; the camera calibration steps are as follows: 1) Capture approximately 20 images containing the calibration plate; 2) Detect corner points on the chessboard image by detecting the positions of corner points in the chessboard image through image processing; 3) Calculate the intrinsic parameter matrix K; The four coordinate systems involved in the camera imaging process are as follows: World coordinate system: A world coordinate system that uses the real world as a reference to describe the pose of objects and cameras in the real physical world. The origin can be chosen independently. A point in the world coordinate system can be represented as (X... w Y w Z w ) T ; Camera coordinate system: The optical center of the camera is taken as the origin, the optical axis of the camera is taken as the z-axis, and the x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system; a point in the camera coordinate system can be represented as (x c y c , z c ) T ; Image coordinate system: The center of the camera's imaging plane is taken as the origin, and the x-axis and y-axis are parallel to two sides of the imaging plane, respectively; a point in the image coordinate system can be represented as (x, y, 1). T ; Pixel coordinate system: The top-left corner of the imaging plane is taken as the origin, and the x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system, respectively; a point in the pixel coordinate system can be represented as (u, v, 1). T ; The relationship between a point in the pixel coordinate system and a point in the world coordinate system is shown in the following formula. Since the size of each square on the captured chessboard is fixed and known, the coordinates of each corner point in the world coordinate system can be obtained and calculated. By using the detected corner point positions in the pixel coordinate system and the coordinates of the corner points in the world coordinate system obtained from the chessboard size, the required intrinsic parameter matrix K can be solved by solving a series of equations based on the formulas corresponding to the aforementioned coordinate systems. The time frame (FrameReadDuration) is obtained based on the camera parameters to complete the exposure of the entire frame image and read the entire frame image data. The IMU data processing module: The IMU sensor and camera sensor are rigidly connected, and can be considered as a single rigid body with the same angular velocity at all points. The camera's attitude change can be indirectly obtained from the raw IMU gyrodata[x,y,z], the gyroscale coefficient gcalse calculated based on the range, and the relative position of the IMU coordinate system and the camera coordinate system. The step of indirectly obtaining the camera attitude quaternion from the raw IMU gyrodata[x,y,z] is handled by the camera attitude calculation module below. The time interval dt = 1 / frequency of the IMU data can be obtained from the sampling frequency frequency. The three-axis angular velocities of the gyroscope can then be obtained. ω x =gyrodata.x*gscale ω y =gyrodata.y*gscale ω z =gyrodata.z*gscale; The trajectory smoothing module further includes: Current camera attitude calculation: The camera attitude is calculated by integrating the camera's angular velocity based on the IMU data, and the camera's rotation quaternion is obtained. The three angular velocity components measured by the three-axis gyroscope can be combined into an angular velocity vector. This can be understood as the camera rotating around this angular velocity vector, with the angle of rotation being the integral of the magnitude of the angular velocity vector. The camera's pose quaternion can be obtained by integrating this data. Camera's real motion attitude trajectory: The camera's real motion attitude trajectory, real_quaternion_trajectory, is saved based on the IMU timestamp (IMU_timestamp) and the camera's attitude quaternion calculated from the IMU data. Saving here refers to the process of storing the IMU timestamp (IMU_timestamp) and the camera's attitude quaternion calculated from the IMU data as a set of data into a deque queue as the real motion attitude trajectory, real_quaternion_trajectory. The smoothed virtual trajectory: Based on the smoothing coefficient α (α∈[0,1]), spherical linear interpolation is performed on the quaternions of adjacent frames to obtain the smoothed quaternion V, which is then saved as the smoothed virtual quaternion trajectory virtual_quaternion_trajectory. This saving process involves storing the current frame image timestamp and pose quaternion V as a set of data into a deque queue as the virtual quaternion trajectory virtual_quaternion_trajectory. The spherical linear interpolation function slerp() calculates t based on the above data and formula. i The quaternion at time α; where α is calculated by the formula below. quaternion1 is the quaternion pose at time t1, quaternion2 is the quaternion pose at time t2, and α is the current t. i The proportion of quaternions at time points during the time interval from t1 to t2; The intra-frame motion compensation module further includes: Since the camera sensor uses rolling shutter exposure, the camera's spatial orientation changes during camera movement, resulting in different camera orientations for each row of pixels during exposure, which can easily cause rolling shutter effect; intra-frame motion compensation is performed to address this issue. Let v be the row in the image corresponding to point p, mHeight be the total number of rows in the image, FrameReadDuration be the time required to read the complete frame row by row from top to bottom, and Ti be the timestamp of the middle row of the i-th frame starting to be exposed. The image transformation output module further includes: The cropping magnification matrix Trimscale_Matrix of the image is calculated based on the cropping ratio, and the cropping magnification compensation transformation matrix of the image is also calculated. MatrixFrameTransform=MatrixFrame*Trimscale_Matrix; Based on the final motion compensation matrix queue, the image is motion compensated pixel by pixel and then cropped and enlarged to output a stabilized real-time video stream.

7. A real-time video stabilization system based on IMU for intra-frame motion compensation according to claim 6, characterized in that, The intra-frame motion compensation module includes the following steps:

0. Start intra-frame motion compensation; 1. Initialize the line exposure delay time to mLineDelay, where mLineDelay = FrameReadDuration / mHeight; 2. Calculate the time difference timeDifference_v between the current v-th row and the middle row mHeight / 2 in the current frame image, timeDifference_v=mLineDelay*(v-mHeight / 2); 3. Obtain the smoothed pose at the moment the camera's middle row begins exposure as the smoothed virtual pose of the current frame, VirtualPose, VirtualPose = virtual_quaternion_trajectory.get_interpolate(Ti); To obtain the pose at the current moment in the pose trajectory, use get_interpolate(Ti) to find the virtual camera poses V1 and V2 corresponding to the two timestamps t1 and t2 adjacent to Ti in the virtual camera pose trajectory virtual_quaternion_trajectory. Calculate the scaling factor s = (Ti-t1) / (t1-t2) at the current moment to obtain the virtual camera pose VirtualPose corresponding to the video frame at time Ti, VirtualPose = slerp(V1, V2, s); Then, based on time timeDifference_v+Ti, interpolate from the real camera pose trajectory real_quatenion_trajectory to obtain the real pose RealPose starting from the v-th row of the i-th frame image and the inverse Realposelnv of the real pose, which is the pose transformation matrix that will transform the current real pose to the initial pose. RealPose = real_quaternion_trajectory.get_interpolate(timeDifference_v+Ti).

4. Obtain the motion compensation matrix MatrixCorrect, which compensates the image in this row to the virtual pose VirtualPose of the image in this frame. MatrixCorrect = RealPoselnv * VirtualPose; 5. After obtaining the motion compensation matrix of the image in this row of the image frame, calculate the image frame transformation matrix MatrixFrame based on the motion compensation matrix and the camera intrinsic parameter matrix K. Then, compensate and correct the rolling shutter effect for each image data P(u,v) in this row of the image. MatrixFrame = K * MatrixCorrect * K_inv; 6. The computation of motion compensation matrix for line-by-line compensated image data is relatively large. It can be divided into n regions according to the image height, and it is assumed that the images in the same region have the same motion vector, so as to obtain n motion compensation matrices of the image.

7. After intra-frame stabilization, the frame borders of the image also change, which may cause a black border to appear in the central region of interest. Black border removal is then necessary. Intra-frame black border removal uses an intra-frame black border removal module to ensure that meaningless black areas do not appear within the image after motion compensation while preserving as much motion compensation effect as possible. This involves calculating trim_width and trim_height based on the trim ratio trim_ratio (trim_ratio∈(0.05,0.15)), cropping and retaining the central region of interest (the middle square area), then enlarging it to its original size to remove the black borders from the stabilized image. trim_width=mWidth*trim_ratio trim_height=mHeight*trim_ratio.

8. A real-time video stabilization system based on IMU for intra-frame motion compensation according to claim 7, characterized in that, The intra-frame black border removal module further includes: Input the initial motion compensation queue to remove black borders; Motion compensation matrix queue reduction: The current motion compensation matrix queue is calculated by interpolation based on the current motion compensation coefficients; Image frame border calculation: Previously, the obtained image was a quadrilateral after perspective transformation of the entire frame. Now, the image border after intra-frame compensation is an irregular polygon. The compensated image frame border needs to be calculated based on the number of layers. Black border detection: Calculate the relationship between the compensated image frame border and the central ROI region. If the central ROI region is within the compensated image frame border, no black border appears, and the current motion compensation coefficient can be output. Conversely, if the central ROI region is not within the compensated image frame border, a black border appears, and the motion compensation coefficient needs to be reduced until a motion compensation coefficient without black borders is obtained. Obtain and output the motion compensation coefficients that do not show black edges after compensation in the current frame compensation matrix queue, and calculate the final motion compensation matrix queue.

Images