A binocular camera stabilization method

Abstract

This invention relates to a method for stabilizing a binocular camera, comprising the following steps: installing an inertial measurement unit (IMU) on each camera; calculating the extrinsic transformation matrix from the right camera coordinate system to the left camera coordinate system after calibrating the binocular camera; obtaining the transformation matrix from the IMU coordinate system to the corresponding camera coordinate system after calibrating each camera and its IMU; calculating the pose transformation matrices of the left and right cameras, and the pose transformation matrices of the left and right camera IMUs, based on image frames captured by the binocular camera at two different times and measurement data collected by the IMU at two different times; transforming all pose transformation matrices to the left camera coordinate system and calculating the fusion transformation matrix; determining whether jitter occurs between the two different times based on the pose transformation matrix, and performing stabilization operation using the fusion transformation matrix. This invention can effectively ensure the stability of the output images from the binocular camera.

Description

Technical Field

[0001] This invention relates to the field of binocular camera imaging technology, and in particular to a binocular camera stabilization method. Background Technology

[0002] Camera stabilization systems can significantly improve the image quality captured by a camera. With the rapid development of various mobile shooting devices, robots, and other fields, people have increasingly higher requirements for the image quality captured by cameras. Traditional stabilization methods mainly include electronic stabilization and mechanical stabilization. Electronic stabilization mainly maintains the stability of the final output image by cropping and transforming the image; mechanical stabilization, on the other hand, uses devices such as motors to compensate for camera displacement and reduce camera shake.

[0003] Current research has yielded numerous studies and applications of electronic stabilization systems for monocular cameras. However, research on stabilization techniques for binocular cameras is relatively limited. Given the high requirements of binocular cameras for precise alignment of left and right images, electronic stabilization methods for monocular cameras are often difficult to directly transfer and apply to binocular camera systems. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for stabilizing binocular cameras, which can effectively ensure the stability of the output images of binocular cameras.

[0005] The technical solution adopted by this invention to solve its technical problem is: to provide a binocular camera stabilization method, comprising the following steps:

[0006] An inertial measurement unit is installed on each camera of the binocular camera system;

[0007] The binocular cameras are calibrated, and the extrinsic transformation matrix P, which transforms the coordinate system of the right camera to the coordinate system of the left camera, is calculated based on the obtained intrinsic parameter matrices of the left and right cameras. I ;

[0008] Each camera and its inertial measurement unit of the binocular camera are calibrated to obtain the transformation matrix P from the coordinate system of the left camera's inertial measurement unit to the coordinate system of the left camera. UL And the transformation matrix P from the right camera inertial measurement unit coordinate system to the right camera coordinate system. UR ;

[0009] Based on the image frames captured by the binocular camera at two different times, the pose transformation matrix P of the left camera is calculated. IL1 The pose transformation matrix P of the right camera IR1 ;

[0010] Based on the measurement data collected by the inertial measurement unit at the two different times, the pose transformation matrix P of the left camera inertial measurement unit is calculated. UL1P, the pose transformation matrix of the right camera inertial measurement unit UR1 ;

[0011] The pose transformation matrix P IL1 The pose transformation matrix P IR1 The pose transformation matrix P UL1 and the pose transformation matrix P UR1 Mapping these coordinates onto the left camera coordinate system yields the transformation matrix P of the left camera. FIL The transformation matrix P of the right camera FIR The transformation matrix P of the left camera inertial measurement unit FUL The transformation matrix P of the right camera inertial measurement unit FUR ;

[0012] If the transformation matrix P FIL Transformation matrix P FIR Transformation matrix P FUL and transformation matrix P FUR All are less than the transformation threshold matrix P Thre If the corresponding set parameters are used, then it is considered that a jitter occurred between the two different times.

[0013] Furthermore, based on the image frames captured by the binocular cameras at two different times, the pose transformation matrix P of the left camera is calculated. IL1 The pose transformation matrix P of the right camera IR1 ,include:

[0014] Based on the first image frames I recorded by the left camera at the two different times... L1 Second image frame I L2 The pose transformation matrix P of the left camera is calculated. IL1 ;

[0015] Based on the first image frames I recorded by the right camera at the two different times... R1 Second image frame I R2 The pose transformation matrix P of the right camera is calculated. IR1 .

[0016] Furthermore, based on the measurement data collected by the inertial measurement unit at the two different times, the pose transformation matrix P of the left camera inertial measurement unit is calculated. UL1 P, the pose transformation matrix of the right camera inertial measurement unit UR1 ,include:

[0017] The first measurement data U recorded by the left camera inertial measurement unit at the two different times is based on the data. L1 Second measurement data U L2The pose transformation matrix P of the left camera inertial measurement unit is calculated. UL1 ;

[0018] The first measurement data U recorded by the right camera inertial measurement unit at the two different times is based on the data. R1 Second measurement data U R2 The pose transformation matrix P of the right camera inertial measurement unit is calculated. UR1 .

[0019] Furthermore, it also includes:

[0020] The transformation matrix P is calculated using the set weight parameters. FIL Transformation matrix P FIR Transformation matrix P FUL and transformation matrix P FUR The weighted sum is used as the fusion transformation matrix P F ;

[0021] If it is assumed that jitter occurred between the two different times, then the fusion transformation matrix P is used. F Stabilize the output image.

[0022] Furthermore, the use of the fusion transformation matrix P F Stabilization operations are performed on the output image, including:

[0023] The left image is the output after calculating the stabilization operation. Among them, K L This is the intrinsic parameter matrix of the left camera;

[0024] The right figure shows the output after calculating the stabilization operation. Among them, K R This is the intrinsic parameter matrix of the right camera.

[0025] Furthermore, the transformation matrix P of the left camera FIL =P IL1 .

[0026] Furthermore, the transformation matrix P of the right camera FIR =P I *P IR1 .

[0027] Furthermore, the transformation matrix P of the left camera inertial measurement unit FUL =P UL *P UL1 .

[0028] Furthermore, the transformation matrix P of the right camera inertial measurement unit FUR =P I *P UR *PUR1 .

[0029] Furthermore, the left and right cameras of the binocular camera share a common field of view.

[0030] Beneficial effects

[0031] By adopting the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art: The present invention installs an inertial measurement unit on each camera of the binocular camera, and solves the coordinate transformation matrix after calibrating the camera and the inertial measurement unit. Then, it solves the pose transformation matrix using image frames at different times and inertial measurement. After mapping all pose transformation matrices to the left camera coordinate system, it calculates the weighted sum to obtain the fusion transformation matrix. Finally, it judges the image frame jitter amplitude based on the fusion transformation matrix, and then decides whether to perform stabilization operation based on the jitter amplitude. This solves the stability problem of the binocular camera system under the requirement of precise alignment of left and right images. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the binocular camera electronic stabilization device according to an embodiment of the present invention;

[0033] Figure 2 This is a flowchart of an embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0035] The present invention relates to an electronic stabilization method for a stereo camera, wherein the embodiment employs, as follows: Figure 1 The illustrated binocular camera comprises two or more cameras with fixed relative positions and a shared field of view. Each camera is equipped with an inertial measurement unit (IMU), and the relative positions of the two IMUs are also fixed. Each frame of image acquired by the cameras and the linear and angular velocity data collected by the IMUs carry timestamp information. A synchronization trigger signal ensures that the timestamp information of the images output by the two cameras and the linear and angular velocity data output by the two IMUs are identical.

[0036] like Figure 2 As shown, the specific process is as follows:

[0037] S1: The stereo cameras were calibrated, and the intrinsic parameter matrix K of the left camera was obtained. L , and the intrinsic parameter matrix K of the right camera RAnd the extrinsic transformation matrix from the right camera image to the left camera is P. I .

[0038] S2: Obtain extrinsic parameters for the camera and its corresponding IMU through calibration, where the transformation matrix from the left IMU to the left camera is P. UL The transformation matrix from the right IMU to the right camera is P. UR .

[0039] S3: Read and record the first frame of the left camera image I L1 And right camera image I R1 and left IMU data U L1 and right IMU data U R1 The final left camera image output in frame 1 is I. LO1 =I L1 The image from the right camera is I. RO1 =I R1 .

[0040] S4: Read and record the second frame of the left camera image I L2 And right camera image I R2 and left IMU data U L2 and right IMU data U R2 .

[0041] S5: via I L1 and I L2 Calculate the pose transformation matrix P between two frames from the left camera. IL1 And through I R1 and I R2 Calculate the pose transformation matrix P between the two frames of the right camera. IR1 .

[0042] S6: Via U L1 and U L2 Calculate the pose transformation matrix P between the two frames of the left IMU. UL1 And through U R1 and U R2 Calculate the pose transformation matrix P between the two frames of the right IMU. UR1 .

[0043] S7: Unify all pose transformation matrices to the coordinates of the left camera. The transformation matrix P corresponding to the left camera... FIL =P IL1 The transformation matrix P corresponding to the right camera FIR =P I *P IR1 ; The transformation matrix P corresponding to the left IMU FUL =P UL *P UL1 ; The transformation matrix P corresponding to the right IMUFUR =P I *P UR *P UR1 .

[0044] S8: Calculate the final fusion transformation matrix P F =k1*P FIL +k2*P FIR +k3*P FUL +k4*P FUR Where k1, k2, k3, and k4 are specific gravity parameters, satisfying k1+k2+k3+k4=1, and all are greater than or equal to 0.

[0045] S9: If P F P in FIL P FIR P FUL P FUR All are less than the transformation threshold matrix P Thre The corresponding parameters in [the code] indicate that the frame's jitter is small and no stabilization is needed; the final output image from the left camera is I. LO2 =I L2 The image from the right camera is I. RO2 =I R2 Otherwise, the frame is considered to have excessive jitter and requires stabilization. The final output image on the left is... The final output image on the right is

[0046] S10: Repeat steps S4 to S9 until the algorithm exits.

[0047] It is worth noting that the relative positions of the camera and IMU cannot be changed after the external parameter calibration is completed.

Claims

1. A method for stabilizing a binocular camera, characterized in that, Includes the following steps: An inertial measurement unit is installed on each camera of the binocular camera system; The binocular cameras are calibrated, and the extrinsic transformation matrix P, which transforms the coordinate system of the right camera to the coordinate system of the left camera, is calculated based on the obtained intrinsic parameter matrices of the left and right cameras. I ; Each camera and its inertial measurement unit of the binocular camera are calibrated to obtain the transformation matrix P from the coordinate system of the left camera's inertial measurement unit to the coordinate system of the left camera. UL And the transformation matrix P from the right camera inertial measurement unit coordinate system to the right camera coordinate system. UR ; Based on the image frames captured by the binocular camera at two different times, the pose transformation matrix P of the left camera is calculated. IL1 The pose transformation matrix P of the right camera IR1 ; Based on the measurement data collected by the inertial measurement unit at the two different times, the pose transformation matrix P of the left camera inertial measurement unit is calculated. UL1 P, the pose transformation matrix of the right camera inertial measurement unit UR1 ; The pose transformation matrix P IL1 The pose transformation matrix P IR1 The pose transformation matrix P UL1 and the pose transformation matrix P UR1 Mapping these coordinates onto the left camera coordinate system yields the transformation matrix P of the left camera. FIL The transformation matrix P of the right camera FIR The transformation matrix P of the left camera inertial measurement unit FUL The transformation matrix P of the right camera inertial measurement unit FUR ; If the transformation matrix P FIL Transformation matrix P FIR Transformation matrix P FUL and transformation matrix P FUR All are less than the transformation threshold matrix P Thre If the corresponding set parameters are used, then it is considered that a jitter occurred between the two different times.

2. The method according to claim 1, characterized in that, The pose transformation matrix P of the left camera is calculated based on image frames captured by the binocular camera at two different times. IL1 The pose transformation matrix P of the right camera IR1 ,include: Based on the first image frames I recorded by the left camera at the two different times... L1 Second image frame I L2 The pose transformation matrix P of the left camera is calculated. IL1 ; Based on the first image frames I recorded by the right camera at the two different times... R1 Second image frame I R2 The pose transformation matrix P of the right camera is calculated. IR1 .

3. The method according to claim 2, characterized in that, The pose transformation matrix P of the left camera inertial measurement unit is calculated based on the measurement data collected by the inertial measurement unit at the two different times. UL1 P, the pose transformation matrix of the right camera inertial measurement unit UR1 ,include: The first measurement data U recorded by the left camera inertial measurement unit at the two different times is based on the data. L1 Second measurement data U L2 The pose transformation matrix P of the left camera inertial measurement unit is calculated. UL1 ; The first measurement data U recorded by the right camera inertial measurement unit at the two different times is based on the data. R1 Second measurement data U R2 The pose transformation matrix P of the right camera inertial measurement unit is calculated. UR1 .

4. The method according to claim 3, characterized in that, Also includes: The transformation matrix P is calculated using the set weight parameters. FIL Transformation matrix P FIR Transformation matrix P FUL and transformation matrix P FUR The weighted sum is used as the fusion transformation matrix P F ; If it is assumed that jitter occurred between the two different times, then the fusion transformation matrix P is used. F Stabilize the output image.

5. The method according to claim 4, characterized in that, The use of the fusion transformation matrix P F Stabilization operations are performed on the output image, including: The left image is the output after calculating the stabilization operation. Among them, K L This is the intrinsic parameter matrix of the left camera; The right figure shows the output after calculating the stabilization operation. Among them, K R This is the intrinsic parameter matrix of the right camera.

6. The method according to claim 1, characterized in that, The transformation matrix P of the left camera FIL =P IL1 .

7. The method according to claim 1, characterized in that, The transformation matrix P of the right camera FIR =P I *P IR1 .

8. The method according to claim 1, characterized in that, The transformation matrix P of the left camera inertial measurement unit FUL =P UL *P UL1 .

9. The method according to claim 1, characterized in that, The transformation matrix P of the right camera inertial measurement unit FUR =P I *P UR *P UR1 .

10. The method according to claim 1, characterized in that, The left and right cameras of the binocular camera share a common field of view.

Images