IMU (Inertial Measurement Unit)-based spherical video anti-shake image stabilization method
By calculating the camera motion trajectory using IMU data and combining it with spherical mapping technology, the problems of background dragging and field of view loss in video stabilization were solved, and stable video processing was achieved under low-light conditions.
Patent Information
- Application Number
- CN202410770227.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-14
- Publication Date
- 2025-12-16
AI Technical Summary
Existing video stabilization methods suffer from severe background dragging when large objects are moving, inaccurate key point detection, loss of field of view due to distortion correction of wide-angle lenses, and poor performance of digital stabilization methods in low light conditions.
The camera motion trajectory is calculated using IMU data. Low-pass filtering smoothing and spherical mapping techniques are combined with cropping and rotation matrix processing to eliminate jitter and retain a large field of view, avoiding distortion correction.
It achieves stable video footage under low-light conditions, avoids background dragging, maintains a large field of view, and improves the accuracy and efficiency of video stabilization.
Smart Images
Figure CN121151684A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of video stabilization technology, and specifically relates to a method for IMU-based spherical video stabilization. Background Technology
[0002] Existing video stabilization technologies include mechanical stabilization, optical stabilization, and digital stabilization. Mechanical stabilization detects camera platform shake using devices such as gyroscope sensors and then adjusts the camera platform system to stabilize the image. This method offers good stabilization results but is energy-intensive, bulky, complex to operate, and has high maintenance costs. Optical stabilization compensates for unstable images using optical components. This method offers high stabilization accuracy but is also complex to operate and has high maintenance costs. Digital stabilization integrates electronics, computer science, and image processing technologies to achieve image stabilization, offering advantages such as high accuracy, low power consumption, and small size. However, most current digital image stabilization methods use key point detection, then motion estimation via optical flow, then smoothing filtering to smooth the motion trajectory, and finally motion compensation to stabilize the shaky image. But this method relies on key point detection. If the image is blurry, the number of key points detected will decrease, thus reducing the stabilization effect. Or when a large area of the object being filmed is moving, background dragging will also occur. In this case, the algorithm cannot completely distinguish whether the object in the image is moving or the camera is moving, which may lead to misjudgment.
[0003] In addition, wide-angle lenses with distortion usually need to go through a distortion correction module to correct the distortion of the video, and then the image after distortion correction is stabilized. However, this will cause a loss of the field of view of the wide-angle lens, making the effective field of view of the stabilized video smaller.
[0004] In addition, commonly used existing technologies include:
[0005] 1. IMU: Generally refers to a 6-axis sensor, which internally contains a 3-axis gyroscope and a 3-axis accelerometer. The 3 axes represent the three coordinate axes in the XYZ plane. The 3-axis gyroscope measures the angular velocity on each axis, with an accuracy typically in ° / s, meaning the number of degrees rotated per second in this direction. The 3-axis accelerometer measures the gravitational acceleration on each axis. For example, when the sensor is placed horizontally on a horizontal surface, theoretically it only experiences gravitational acceleration in the negative z-axis direction, with a magnitude of 9.8 m / s^2.
[0006] 2. Video Stabilization: Based on motion estimation using IMU data between consecutive video images, and then performing motion filtering and motion compensation processing on each frame of the video to obtain stable video frame images.
[0007] 3. Latitude and longitude unfolding algorithm: Through mathematical formula conversion, latitude and longitude coordinates on the sphere are converted into planar coordinates. Summary of the Invention
[0008] In order to solve the above problems, the purpose of this application is:
[0009] 1. This method addresses the issue of unstable video footage caused by camera shake. It directly utilizes data from gyroscopes and accelerometers to calculate the camera's motion trajectory, thus avoiding inaccurate motion estimation due to background drag caused by the movement of large objects. It also reduces the impact of image noise on feature point matching methods at night.
[0010] 2. For wide-angle cameras with distortion, this method does not require distortion correction of the distorted video frames, nor does it require camera intrinsic parameter calibration. It can also preserve a larger field of view for the stabilized video as much as possible.
[0011] Specifically, the present invention provides a method for image stabilization and image fixation based on IMU spherical video, the method comprising the following steps:
[0012] S1, raw video frames, acquire 6-axis sensor IMU data from gyroscope and accelerometer, and integrate the IMU data to obtain the camera's real-time motion trajectory;
[0013] S2, perform low-pass filtering to smooth the IMU data, that is, use low-pass filtering to smooth the real motion trajectory of the camera to obtain the virtual motion trajectory after the camera motion is smoothed.
[0014] S3, Calculate the rotation matrix based on the IMU data after filtering and smoothing in step S2;
[0015] S4, convert to a sphere, which means mapping the distorted two-dimensional video frame onto a three-dimensional sphere in space;
[0016] S5. Use the cropping method to crop the rotated image, and then determine whether the cropped image has black borders. If there are no black borders, convert the rotation quaternion into a rotation matrix.
[0017] If the cropped image has black borders, the rotation quaternion is reduced by gradient descent. Then, it is checked whether the rotated image still has black borders. If black borders still exist, the rotation quaternion is reduced until there are no black borders. Finally, the rotation quaternion is converted into a rotation matrix.
[0018] S6, rotate the sphere, that is, use the rotation quaternion in step S3 to rotate the image on the sphere;
[0019] S7, to obtain image stabilization.
[0020] Step S1 further includes:
[0021] S1.1, acquire camera C at fixed time intervals. a In t x Video stream image data at any given moment I a (t x Assuming the initial video frame timestamp is t1 and the video frame rate is 30 frames per second, which is equivalent to a time interval of 0.03333 seconds between every two video frames, then the timestamp of the second video frame is t2, and t2 = t1 + 0.03333.
[0022] The acquired video frame images and their corresponding timestamps are stored in the video frame queue;
[0023] Then acquire the gyroscope data at each moment.
[0024] The gyroscope uses a frequency of 500 Hz, which is equivalent to 500 gyroscope data points per second. Assuming the timestamp of the first gyroscope data point is ts1, then the timestamp of the second gyroscope data point is ts2, and ts2 = ts1 + 0.002.
[0025] The acquired gyroscope data and corresponding timestamps are stored in the gyroscope data queue;
[0026] S1.2, Integrate the angular velocity data from the gyroscope to obtain the camera C. a The true orientation and attitude of the camera at time t can be obtained using a gyroscope. a Angular velocity in the X, Y, and Z directions;
[0027] The camera C can be obtained from the angular velocities in the three directions. a The angles of rotation around the X, Y, and Z axes are called Euler angles. These Euler angles are then converted into a quaternion p, where a quaternion can represent a vector in three-dimensional space. This p is the camera C. a The actual motion posture at time t is calculated as follows:
[0028]
[0029] Where α, β, and γ represent the angles of rotation about the X, Y, and Z axes, respectively.
[0030] Step S2 includes:
[0031] The actual motion direction of the camera is smoothed using a low-pass filter, the Slerp algorithm, to obtain a new quaternion r. Assuming p1 and p2 represent two adjacent gyroscope data points at timestamps ts1 and ts2, this quaternion r represents the virtual camera pose after filtering at time ts1.
[0032]
[0033] Where θ is the angle between the three-dimensional vectors represented by quaternions p1 and p2, and s represents the smoothing coefficient, which ranges from [0,1]. The closer s is to 0, the smoother the filtering effect.
[0034] Step S2 further includes:
[0035] S2.1, spherical linear interpolation is used to obtain the timestamp t. x The corresponding virtual camera pose quaternion;
[0036] Suppose that the timestamp t1 of the first video frame in the video frame queue cannot be directly found in the virtual camera pose queue, so it is necessary to find the approximate position of t1 in the virtual camera pose timestamp queue. Suppose that t1 is between ts2 and ts3, the virtual camera pose at time t1 is calculated by spherical linear interpolation.
[0037] The spherical linear interpolation method includes:
[0038] Assuming the virtual camera poses at timestamps ts2 and ts3 in the virtual camera pose queue are r2 and r3 respectively, the quaternion rs1 of the virtual camera pose corresponding to the video frame at time t1 is obtained by linear interpolation:
[0039] rs1 = r2*a + r3*(1-a)
[0040] Where 'a' is calculated using timestamps ts2, ts3, and t1, 'a' represents the correlation coefficient between time t1 and timestamps ts2 and ts3, and 'a' represents the weight coefficient of the data at time t1 in relation to the data at timestamps ts2 and ts3.
[0041] a = (t1 - ts2) / (ts3 - ts2);
[0042] S2.2, using the spherical linear interpolation method described above, the same interpolation is applied to the real camera pose queue, and the camera C at time t1 can be obtained. a The true pose quaternion qs1.
[0043] 10. The method for image stabilization based on IMU spherical video according to claim 4, wherein step S3 further includes:
[0044] S3.1 Next, the motion compensation quaternion quat1 of the camera at time t1 is calculated using the camera's true pose quaternion qs1 and the camera's virtual pose quaternion rs1:
[0045]
[0046] in Let quaternion qs1 and quaternion rs1 be the conjugates of quaternion rs1.
[0047] qs1=(q w q x q y q z );
[0048] rs1 = (p w p x p y p z );
[0049] Then the conjugate of rs1 for:
[0050]
[0051] Calculating rotations in three-dimensional space using quaternions avoids the gimbal lock problem associated with Euler angles.
[0052] S3.2, convert the motion compensation quaternion into a rotation matrix. Specifically, let the motion compensation quaternion be quatl = (θ, x, y, z). Then the formula for calculating the rotation matrix R is:
[0053]
[0054] Step S4 further includes:
[0055] S4.1 unfolds the video frames from the wide-angle lens according to latitude and longitude.
[0056] Assuming a point (a, b) in a given wide-angle video frame, a spherical radius r1 of 960, a field of view (FOV) of 180 degrees, and a latitude / longitude rotation matrix rot:
[0057]
[0058] S4.2, First, to calculate the position of point (a, b) on the latitude and longitude unfolded map, it is necessary to first calculate the latitude and longitude of that point on the sphere, where... θ2 represents longitude, and θ2 represents latitude.
[0059]
[0060]
[0061] S4.3, then, calculate the three-dimensional coordinates x2, y2, z2 in the spatial solid using latitude and longitude:
[0062] z² = cos(θ² * π / 180)
[0063]
[0064]
[0065] Converting two-dimensional video images onto a three-dimensional sphere allows for rotation of the image at any angle in three-dimensional space, thus better matching the handheld camera's movement and angle during shooting, and also better eliminating camera shake during shooting.
[0066] Step S5 further includes:
[0067] S5.1, the three-dimensional coordinates in step S4 are obtained by rotation:
[0068]
[0069] Where temp represents the intermediate vector of a 3x1 matrix, the coordinates (m, n) of the point (a, b) on the original video frame are finally calculated on the latitude and longitude unfolded map. The latitude and longitude unfolding formula is:
[0070]
[0071]
[0072] S5.2, determine whether the spherical unfolded image of the video frame has been rotated by the rotation matrix; set a suitable cropping ratio α = 0.15, and then use the cropping ratio and the size of the original video frame to obtain a cropping frame. Assuming that the length and width of the original video frame are w = 1920 and h = 1080 respectively, the four coordinates of the cropping window can be calculated as (288, 162), (288, 918), (1632, 918) and (1632, 162); the purpose of setting the cropping frame is to give the original jittery video frame a certain smooth and stable space. The perspective transformation matrix T of cropping scaling can be calculated by the scaling ratio.
[0073] The four vertices of the cropping frame can be used to obtain four points on the latitude and longitude map using the latitude and longitude expansion formula above;
[0074] Then, it is determined that the four vertices of the original video frame are mapped onto the latitude and longitude unfolded map through the latitude and longitude unfolded mapping relationship. Then, the frame is rotated by a rotation matrix. Then, it is determined whether the rotated video frame can contain the four vertices. If it can contain the four vertices, the rotation matrix does not need to be changed. If it cannot cover the four vertices, the rotation amount is reduced so that the four vertices can fall into the latitude and longitude unfolded map of the video frame.
[0075] Step S6 further includes:
[0076] Assuming the points (u, v) on the latitude and longitude unfolded map are known, and the perspective transformation matrix for clipping and scaling is a 3x3 matrix T, first perform a perspective transformation on the points on the sphere to obtain the scaled points (i, j) on the sphere:
[0077]
[0078] Where proj is a 3x1 intermediate vector:
[0079] i = proj(0) / proj(2)
[0080] j = proj(1) / proj(2)
[0081] The coordinates on the latitude and longitude unfolded map can be converted to three-dimensional points x, y, z on the sphere:
[0082] θ=j*π / r
[0083]
[0084] Where r represents the radius of the sphere, r = 1080. Continue solving for x, y, and z:
[0085]
[0086]
[0087] z = cosθ;
[0088] The gyroscope angular velocity data is converted into quaternions. By integrating the quaternions, the pose of the camera in the current frame in 3D space can be obtained. Then, the rotation matrix R of the current frame can be calculated. The calculation process of the rotation matrix is shown in steps S3 and S5. By rotating the entire sphere using the rotation matrix, new 3D coordinate points x1, y1, z1 can be obtained.
[0089]
[0090] Spherical coordinate system transformation:
[0091] θ1=arc cos(z1)*180 / π
[0092]
[0093] Coordinate transformation from sphere to fisheye:
[0094] r1 = r * θ1 / (180 / 2)
[0095] This r1 represents the spherical radius of the fisheye lens, from which the point (u, v) on the latitude and longitude map can be calculated, corresponding to the point (u′, v′) on the fisheye image:
[0096]
[0097] This allows us to obtain a point-to-point mapping relationship between the latitude and longitude unfolded map and the fisheye map;
[0098] The mapping relationship described above enables image stabilization of the current video frame.
[0099] Therefore, the advantage of this application is:
[0100] 1. The camera's motion pose is calculated using gyroscope and accelerometer data, and the motion is smoothed to reduce or eliminate camera shake during recording, resulting in a relatively stable viewing angle in the captured footage.
[0101] 2. This method can directly use data from gyroscopes and accelerometers to calculate the camera's motion trajectory, thus avoiding the problem of inaccurate motion estimation caused by background dragging due to the movement of large-area objects. At the same time, it can reduce the impact of image noise on feature point matching methods at night.
[0102] 3. For wide-angle cameras with distortion, the method does not require distortion correction of the distorted video frames, nor does it require camera intrinsic parameter calibration. It can also preserve a larger field of view as much as possible for the stabilized video. Attached Figure Description
[0103] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0104] Figure 1 This is a schematic diagram of the motion smoothing filter effect in this application.
[0105] Figure 2 This is a schematic diagram illustrating the timestamp interpolation alignment of video frames and virtual cameras in this application.
[0106] Figure 3 This is a schematic diagram illustrating the timestamp interpolation alignment between video frames and real camera data in this application.
[0107] Figure 4This is a schematic diagram of the actual image captured by the wide-angle lens in this application.
[0108] Figure 5 This is the latitude and longitude unfolded map in this application.
[0109] Figure 6 This is a schematic diagram of the clipping frame in this application.
[0110] Figure 7 This is a schematic diagram showing the position of the cropping frame vertex on the latitude and longitude map in this application.
[0111] Figure 8 This is a schematic diagram of the spherical perspective projection model in this application.
[0112] Figure 9 This is a schematic diagram of the overall process of the method described in this application. Detailed Implementation
[0113] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0114] This invention relates to an improved method for quantization calibration of neural networks. To achieve the above objectives, this invention is implemented through the following technical solution:
[0115] S1, raw video frames, acquire 6-axis sensor IMU data from gyroscope and accelerometer, and integrate the IMU data to obtain the camera's real-time motion trajectory;
[0116] S1.1, acquire camera C at fixed time intervals. a In t x Video stream image data at any given moment I a (t x Assuming the initial video frame timestamp is t1 and the video frame rate is 30 frames per second, which is equivalent to a time interval of 0.03333 seconds between every two video frames, then the timestamp of the second video frame is t2, and t2 = t1 + 0.03333.
[0117] The acquired video frame images and their corresponding timestamps are stored in the video frame queue;
[0118] Then acquire the gyroscope data at each moment.
[0119] The gyroscope uses a frequency of 500 Hz, which is equivalent to 500 gyroscope data points per second. Assuming the timestamp of the first gyroscope data point is ts1, then the timestamp of the second gyroscope data point is ts2, and ts2 = ts1 + 0.002.
[0120] The acquired gyroscope data and corresponding timestamps are stored in the gyroscope data queue;
[0121] S1.2, Integrate the angular velocity data from the gyroscope to obtain the camera C. a The true orientation and attitude of the camera at time t can be obtained using a gyroscope. a Angular velocity in the X, Y, and Z directions;
[0122] The camera C can be obtained from the angular velocities in the three directions. a The angles of rotation around the X, Y, and Z axes are called Euler angles. These Euler angles are then converted into a quaternion p, where a quaternion can represent a vector in three-dimensional space. This p is the camera C. a The actual motion posture at time t is calculated as follows:
[0123]
[0124] Where α, β, and γ represent the angles of rotation about the X, Y, and Z axes, respectively.
[0125] Step S2 includes:
[0126] The actual motion direction of the camera is smoothed using a low-pass filter, the Slerp algorithm, to obtain a new quaternion r. Assuming p1 and p2 represent two adjacent gyroscope data points at timestamps ts1 and ts2, this quaternion r represents the virtual camera pose after filtering at time ts1.
[0127]
[0128] Where θ is the angle between the three-dimensional vectors represented by quaternions p1 and p2, and s represents the smoothing coefficient, with a value range of [0,1]. The effect of motion smoothing is as follows: Figure 1 As shown, the closer s is to 0, the more stable the filtering effect.
[0129] Step S2 further includes:
[0130] S2.1, to obtain the camera pose corresponding to each video frame, the sampling frequency of the gyroscope data is different from the sampling frequency of the video frame, resulting in different timestamps t of the video frame. x The timestamp t may not exist directly in the gyroscope data queue, so spherical linear interpolation is needed to obtain it. x The corresponding virtual camera pose quaternion.
[0131] For example Figure 2As shown, the timestamp t1 of the first video frame in the video frame queue cannot be directly found in the virtual camera pose queue. Therefore, it is necessary to find the approximate position of time t1 in the virtual camera pose timestamp queue. For example, if time t1 is between ts2 and ts3, the virtual camera pose at time t1 can be calculated by spherical linear interpolation.
[0132] Linear interpolation process:
[0133] For example Figure 2 The virtual camera poses corresponding to timestamps ts2 and ts3 in the virtual camera pose queue are shown as r2 and r3, respectively. The virtual camera orientation rs1 corresponding to the video frame at time t1 is obtained by linear interpolation.
[0134] rs1 = r2*a + r3*(1-a);
[0135] Where 'a' is calculated using timestamps ts2, ts3, and t1, 'a' represents the correlation coefficient between time t1 and timestamps ts2 and ts3, and 'a' represents the weight coefficient of the data at time t1 in relation to the data at timestamps ts2 and ts3.
[0136] a = (t1 - ts2) / (ts3 - ts2);
[0137] S2.2, using the linear interpolation method described above to perform the same interpolation on the real camera pose queue, we can get the camera C at time t1. a The true pose quaternion qs1, such as Figure 3 As shown;
[0138] S3, Calculate the rotation matrix based on the IMU data after filtering and smoothing in step S2; further includes:
[0139] S3.1 Next, the motion compensation quaternion quat1 of the camera at time t1 is calculated using the camera's true pose quaternion qs1 and the camera's virtual pose quaternion rs1:
[0140]
[0141] in Let quaternion qs1 and quaternion rs1 be the conjugates of quaternion rs1.
[0142] qs1=(q w q x ,q y q z );
[0143] rs1 = (p w p x py p z );
[0144] Then the conjugate of rs1 for:
[0145]
[0146] Calculating rotations in 3D space using quaternions avoids the gimbal lock problem associated with Euler angles. Euler angles include yaw, pitch, and roll. When rotating, we set a rotation order, sorting these three angles as follows: yaw -> pitch -> roll. Rotating yaw affects both the pitch and roll axes, rotating pitch affects the pitch axis, and rotating roll only affects itself. Therefore, a deadlock problem occurs when rotating the roll and yaw axes to the same plane.
[0147] S3.2, convert the motion compensation quaternion into a rotation matrix. Specifically, let the motion compensation quaternion be quatl = (θ, x, y, z). Then the formula for calculating the rotation matrix R is:
[0148]
[0149] S4, conversion to a sphere, that is, mapping the distorted two-dimensional video frame onto a three-dimensional sphere; further including:
[0150] S4.1 unfolds the video frames from the wide-angle lens according to latitude and longitude.
[0151] Assuming a point (a, b) in a given wide-angle video frame, a spherical radius r1 of 960, a field of view (FOV) of 180 degrees, and a latitude / longitude rotation matrix rot:
[0152]
[0153] S4.2, First, to calculate the position of point (a, b) on the latitude and longitude unfolded map, it is necessary to first calculate the latitude and longitude of that point on the sphere, where... θ2 represents longitude, and θ2 represents latitude.
[0154]
[0155]
[0156] S4.3, then, calculate the three-dimensional coordinates x2, y2, z2 in the spatial solid using latitude and longitude:
[0157] z² = cos(θ² * π / 180)
[0158]
[0159]
[0160] Converting two-dimensional video images onto a three-dimensional sphere allows for rotation of the image at any angle in three-dimensional space, thus better matching the handheld camera's movement and angle during shooting, and also better eliminating camera shake during shooting.
[0161] S5: Crops the rotated image using a cropping method, then checks if the cropped image has black borders. If there are no black borders, the rotation matrix calculated in S3 remains unchanged. If black borders exist, the rotation quaternion is reduced using gradient descent. The process continues to check if black borders still exist; if they do, the rotation quaternion is reduced until no black borders remain. Finally, the rotation quaternion is converted into a rotation matrix. Further steps include:
[0162] S5.1, the three-dimensional coordinates x2, y2, z2 in the spatial solid in step S4 are obtained after rotation:
[0163]
[0164] Where temp represents the intermediate vector of a 3x1 matrix, the coordinates (m, n) of the point (a, b) on the original video frame can be calculated on the latitude and longitude unfolded map, as shown in the figure. Figure 5 As shown:
[0165]
[0166]
[0167] S5.2, determine whether the spherical unfolded image of the video frame has been rotated using the rotation matrix, set a suitable cropping ratio α = 0.15, and then use the cropping ratio and the size of the original video frame to obtain a cropping frame. For example, if the original video frame is as follows... Figure 4 As shown, its length and width are w = 1920 and h = 1080, respectively. Therefore, the four coordinates of the cropping window can be calculated as (288, 162), (288, 918), (1632, 918), and (1632, 162). The purpose of setting the cropping frame is to provide a certain smooth and stable space for the original shaky video frames. The perspective transformation matrix T for cropping scaling can be calculated using the scaling factor, as shown below. Figure 6 As shown, the rectangle represents the cropping window. The four vertices of the cropping frame can be used to obtain four points on the latitude and longitude map using the latitude and longitude expansion formula above, as shown... Figure 7As shown in the diagram, the four points are the vertices mapped by the cropping box. Then, the four vertices of the original video frame are mapped onto a latitude and longitude unfolded graph using the latitude and longitude unfolding mapping relationship. Next, a rotation matrix is applied, and then it is determined whether the rotated video frame can contain the four vertices. If it can, no changes to the rotation matrix are needed; otherwise, the rotation amount is reduced so that the four vertices fall within the latitude and longitude unfolded graph of the video frame.
[0168] S6, rotate the sphere, that is, use the rotation quaternion in step S5 to rotate the image on the sphere; step S4 first converts the two-dimensional planar image to the three-dimensional sphere, and step S6 uses the rotation matrix calculated in S5 to rotate the three-dimensional sphere.
[0169] Assuming we have known the points (u, v) on the latitude and longitude unfolded map, and the perspective transformation matrix (3x3 matrix) T for cropping and scaling, we first perform a perspective transformation on the points on the sphere (perspective transformation is an image processing technique used to map points on one plane to corresponding points on another plane, while preserving the straight-line properties of lines in both the original and transformed images), to obtain the scaled points (i, j) on the sphere:
[0170]
[0171] Where proj is a 3x1 intermediate vector:
[0172] i = proj(0) / proj(2)
[0173] j = proj(1) / proj(2)
[0174] The coordinates on the latitude and longitude unfolded map can be converted to three-dimensional points x, y, z on a sphere, such as... Figure 8 As shown:
[0175] θ=j*π / r
[0176]
[0177] Where r represents the radius of the sphere, r = 1080. Continue solving for x, y, and z:
[0178]
[0179]
[0180] z = cosθ;
[0181] The gyroscope angular velocity data is converted into quaternions. By integrating the quaternions, the pose of the camera in the current frame in 3D space can be obtained. Then, the rotation matrix R of the current frame can be calculated. The calculation process of the rotation matrix is shown in S3 and S5. By rotating the entire sphere using the rotation matrix, new 3D coordinate points x1, y1, z1 can be obtained.
[0182]
[0183] Spherical coordinate system transformation:
[0184] θ1=arc cos(z1)*180 / π
[0185]
[0186] Coordinate transformation from sphere to fisheye:
[0187] r1 = r * θ1 / (180 / 2)
[0188] This r1 represents the spherical radius of the fisheye lens, from which we can calculate the point (u′, v′) on the fisheye image corresponding to point (u, v) on the latitude and longitude map:
[0189]
[0190]
[0191] This allows us to obtain a point-to-point mapping from the latitude and longitude unfolded map to the fisheye image, which can be used to stabilize the current video frame.
[0192] S7 finally obtains the image after image stabilization, thus achieving spherical image stabilization.
[0193] In summary, the steps of this method are as follows: Figure 9 As shown:
[0194] S1, raw video frames, acquire 6-axis sensor IMU data from gyroscope and accelerometer, and integrate the IMU data to obtain the camera's real-time motion trajectory;
[0195] S2, perform low-pass filtering to smooth the IMU data, that is, use low-pass filtering to smooth the real motion trajectory of the camera to obtain the virtual motion trajectory after the camera motion is smoothed.
[0196] S3, Calculate the rotation matrix based on the IMU data after filtering and smoothing in step S2;
[0197] S4, convert to a sphere, which means mapping the distorted two-dimensional video frame onto a three-dimensional sphere in space;
[0198] S5 uses a cropping method to crop the rotated image, and then checks whether the cropped image has black borders. If there are no black borders, the rotation matrix calculated in S3 remains unchanged. If the cropped image has black borders, the rotation quaternion is reduced by gradient descent. Then, the rotation quaternion is reduced until there are no black borders. Finally, the rotation quaternion is converted into a rotation matrix.
[0199] S6, rotate the sphere, that is, use the rotation quaternion in step S5 to rotate the image on the sphere;
[0200] S7, to obtain image stabilization.
[0201] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for image stabilization and image fixing based on IMU spherical video, characterized in that, The method includes the following steps: S1, raw video frames, acquire 6-axis sensor IMU data from gyroscope and accelerometer, and integrate the IMU data to obtain the camera's real-time motion trajectory; S2, perform low-pass filtering to smooth the IMU data, that is, use low-pass filtering to smooth the real motion trajectory of the camera to obtain the virtual motion trajectory after the camera motion is smoothed. S3, Calculate the rotation matrix based on the IMU data after filtering and smoothing in step S2; S4, convert to a sphere, which means mapping the distorted two-dimensional video frame onto a three-dimensional sphere in space; S5 uses a cropping method to crop the rotated image, and then checks whether the cropped image has black borders. If there are no black borders, the rotation matrix calculated in S3 remains unchanged. If the cropped image has black borders, the rotation quaternion is reduced by gradient descent. Then, the rotation quaternion is reduced until there are no black borders. Finally, the rotation quaternion is converted into a rotation matrix. S6, rotate the sphere, that is, use the rotation quaternion in step S5 to rotate the image on the sphere; S7, to obtain image stabilization.
2. The method for image stabilization based on IMU spherical video according to claim 1, characterized in that, Step S1 further includes: S1.1, acquire camera C at fixed time intervals. a In t x Video stream image data at any given moment I a (t x Assuming the initial video frame timestamp is t1 and the video frame rate is 30 frames per second, which is equivalent to a time interval of 0.03333 seconds between every two video frames, then the timestamp of the second video frame is t2, and t2 = t1 + 0.03333. The acquired video frame images and their corresponding timestamps are stored in the video frame queue; Then acquire the gyroscope data at each moment. The gyroscope uses a frequency of 500 Hz, which is equivalent to 500 gyroscope data points per second. Assuming the timestamp of the first gyroscope data point is ts1, then the timestamp of the second gyroscope data point is ts2, and ts2 = ts1 + 0.
002. The acquired gyroscope data and corresponding timestamps are stored in the gyroscope data queue; S1.2, Integrate the angular velocity data from the gyroscope to obtain the camera C. a The true orientation and attitude of the camera at time t can be obtained using a gyroscope. a Angular velocity in the X, Y, and Z directions; The camera C can be obtained from the angular velocities in the three directions. a The angles of rotation around the X, Y, and Z axes are called Euler angles. These Euler angles are then converted into a quaternion p, where a quaternion can represent a vector in three-dimensional space. This p is the camera C. a The actual motion posture at time t is calculated as follows: Where α, β, and γ represent the angles of rotation about the X, Y, and Z axes, respectively.
3. The method for image stabilization based on IMU spherical video according to claim 2, characterized in that, Step S2 includes: The actual motion direction of the camera is smoothed using a low-pass filter, the Slerp algorithm, to obtain a new quaternion r. Assuming p1 and p2 represent two adjacent gyroscope data points at timestamps ts1 and ts2, this quaternion r represents the virtual camera pose after filtering at time ts1. Where θ is the angle between the three-dimensional vectors represented by quaternions p1 and p2, and s represents the smoothing coefficient, which ranges from [0,1]. The closer s is to 0, the smoother the filtering effect.
4. The method for image stabilization based on IMU spherical video according to claim 2, characterized in that, Step S2 further includes: S2.1, spherical linear interpolation is used to obtain the timestamp t. x The corresponding virtual camera pose quaternion; Suppose that the timestamp t1 of the first video frame in the video frame queue cannot be directly found in the virtual camera pose queue, so it is necessary to find the approximate position of t1 in the virtual camera pose timestamp queue. Suppose that t1 is between ts2 and ts3, the virtual camera pose at time t1 is calculated by spherical linear interpolation. The spherical linear interpolation method includes: Assuming the virtual camera poses at timestamps ts2 and ts3 in the virtual camera pose queue are r2 and r3 respectively, the quaternion rs1 of the virtual camera pose corresponding to the video frame at time t1 is obtained by linear interpolation: rs1 = r2*a + r3*(1-a) Where 'a' is calculated using timestamps ts2, ts3, and t1, 'a' represents the correlation coefficient between time t1 and timestamps ts2 and ts3, and 'a' represents the weight coefficient of the data at time t1 in relation to the data at timestamps ts2 and ts3. a = (t1 - ts2) / (ts3 - ts2); S2.2, using the spherical linear interpolation method described above, the same interpolation is applied to the real camera pose queue, and the camera C at time t1 can be obtained. a The true pose quaternion qs1.
5. The method for image stabilization based on IMU spherical video according to claim 3, characterized in that, Step S3 further includes: S3.1 Next, the motion compensation quaternion quat1 of the camera at time t1 is calculated using the camera's true pose quaternion qs1 and the camera's virtual pose quaternion rs1: in Let quaternion qs1 and quaternion rs1 be the conjugates of quaternion rs1. qs1=(q w q x ,q y ,q z ); rs1=(p w ,p x ,p y ,p z ): Then the conjugate of rs1 for: Calculating rotations in three-dimensional space using quaternions avoids the gimbal lock problem associated with Euler angles. S3.2, convert the motion compensation quaternion into a rotation matrix. Specifically, let the motion compensation quaternion quat1 = (θ, x, y, z), then the formula for calculating the rotation matrix R is:
6. The method for image stabilization based on IMU spherical video according to claim 4, characterized in that, Step S4 further includes: S4.1 unfolds the video frames from the wide-angle lens according to latitude and longitude. Assuming a point (a, b) in a given wide-angle video frame, a spherical radius r1 of 960, a field of view (FOV) of 180 degrees, and a latitude / longitude rotation matrix rot: S4.2, First, to calculate the position of point (a,b) on the latitude and longitude unfolded map, it is necessary to first calculate the latitude and longitude of that point on the sphere, where... θ2 represents longitude, and θ2 represents latitude. S4.3, then, calculate the three-dimensional coordinates x2, y2, z2 in the spatial solid using latitude and longitude: z² = cos(θ² * π / 180) Converting two-dimensional video images onto a three-dimensional sphere allows for rotation of the image at any angle in three-dimensional space, thus better matching the handheld camera's movement and angle during shooting, and also better eliminating camera shake during shooting.
7. The method for image stabilization based on IMU spherical video according to claim 5, characterized in that, Step S5 further includes: S5.1, the three-dimensional coordinates in step S4 are obtained by rotation: Where temp represents the intermediate vector of a 3x1 matrix, the coordinates (m, n) of the point (a, b) on the original video frame are finally calculated on the latitude and longitude unfolded map. The latitude and longitude unfolding formula is: S5.2, determine whether the spherical unfolded image of the video frame has been rotated by the rotation matrix; set a suitable cropping ratio α = 0.15, and then use the cropping ratio and the size of the original video frame to obtain a cropping frame. Assuming that the length and width of the original video frame are w = 1920 and h = 1080 respectively, the four coordinates of the cropping window can be calculated as (288, 162), (288, 918), (1632, 918) and (1632, 162); setting the cropping frame is to give the original jittery video frame a certain smooth and stable space. The perspective transformation matrix T of cropping scaling can be calculated through the scaling ratio. The four vertices of the cropping frame can be used to obtain four points on the latitude and longitude map using the latitude and longitude expansion formula above; Then, it is determined that the four vertices of the original video frame are mapped onto the latitude and longitude unfolded map through the latitude and longitude unfolded mapping relationship. Then, the frame is rotated using a rotation matrix. Then, it is determined whether the rotated video frame can contain the four vertices. If it can contain the four vertices, the rotation matrix does not need to be changed. If it cannot cover the four vertices, the rotation amount is reduced so that the four vertices can fall into the latitude and longitude unfolded map of the video frame.
8. The method for image stabilization based on IMU spherical video according to claim 6, characterized in that, Step S6 includes: Assuming the points (u, v) on the latitude and longitude unfolded map are known, and the perspective transformation matrix for clipping and scaling is a 3x3 matrix T, first perform a perspective transformation on the points on the sphere to obtain the scaled points (i, j) on the sphere: Where proj is a 3x1 intermediate vector: i = proj(0) / proj(2) j = proj(1) / proj(2) The coordinates on the latitude and longitude unfolded map can be converted to three-dimensional points x, y, z on the sphere: 0θ=j*TT / r Where r represents the radius of the sphere, r = 1080, and we continue to solve for x, y, z: z = cosθ; The gyroscope angular velocity data is converted into quaternions. By integrating the quaternions, the pose of the camera in the current frame in 3D space can be obtained. Then, the rotation matrix R of the current frame can be calculated. The calculation process of the rotation matrix is shown in steps S3 and S5. By rotating the entire sphere using the rotation matrix, new 3D coordinate points x1, y1, z1 can be obtained. Spherical coordinate system transformation: θ1=arccos(z1)*180 / π Coordinate transformation from sphere to fisheye: r1 = r * θ1 / (180 / 2) This r1 represents the spherical radius of the fisheye lens, from which the point (u, v) on the latitude and longitude map can be calculated, corresponding to the point (u′, v′) on the fisheye image: This allows us to obtain a point-to-point mapping relationship between the latitude and longitude unfolded map and the fisheye map; The mapping relationship described above enables image stabilization of the current video frame.
9. The method for image stabilization based on IMU spherical video according to claim 7, characterized in that, In step S7, the stabilized video frame can be obtained through the mapping relationship calculated in S6.