Binocular camera stability augmentation method

Abstract

The invention relates to a binocular camera stability augmentation method. The method comprises the following steps: installing an inertial measurement unit on each camera; calibrating the binocular camera and then calculating to obtain an external parameter transformation matrix for converting a right camera coordinate system into a left camera coordinate system; calibrating each camera of the binocular camera and the inertial measurement unit thereof to obtain a transformation matrix for converting the coordinate system of the inertial measurement unit to the coordinate system of the corresponding camera; according to image frames shot by the binocular camera at two different moments and measurement data collected by the inertial measurement unit, a pose transformation matrix of the left camera and the right camera and a pose transformation matrix of the inertial measurement unit of the left camera and the right camera are solved; all the pose transformation matrixes are converted into a left camera coordinate system, a fusion transformation matrix is calculated, whether jitter occurs between the two different moments is judged based on the pose transformation matrixes, and stability augmentation operation is conducted through the fusion transformation matrix. The stability of the output image of the binocular camera can be effectively ensured.

Description

Technical Field

[0001] The present invention relates to the field of binocular camera imaging technology, and in particular to a binocular camera stabilization method. Background Art

[0002] The camera stabilization system can significantly improve the image quality of the camera. With the rapid development of various mobile shooting equipment, robots and other fields, people's requirements for the quality of camera images are getting higher and higher. Traditional stabilization methods mainly include electronic stabilization and mechanical stabilization. Electronic stabilization mainly maintains the stability of the final output image by cropping and transforming the image; mechanical stabilization uses motors and other devices to reversely compensate for the displacement of the camera and reduce camera shake.

[0003] In current research, there have been many studies and applications of monocular camera electronic stabilization systems. However, there are relatively few studies on stabilization technology for binocular cameras. Given the high requirements of binocular cameras for precise alignment of left and right images, the electronic stabilization method of monocular cameras is often difficult to directly transfer and apply to binocular camera systems. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a binocular camera stabilization method, which can effectively ensure the stability of the binocular camera output image.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a binocular camera stabilization method, comprising the following steps:

[0006] Install an inertial measurement unit on each camera of the stereo camera;

[0007] The binocular camera is calibrated, and the external parameter transformation matrix P of the right camera coordinate system to the left camera coordinate system is calculated based on the obtained internal parameter matrices of the left and right cameras. I ;

[0008] Calibrate each camera and its inertial measurement unit of the binocular camera to obtain the transformation matrix P of the left camera inertial measurement unit coordinate system to the left camera coordinate system UL , and the transformation matrix P from the right camera inertial measurement unit coordinate system to the right camera coordinate system UR ;

[0009] According to the image frames taken by the binocular camera at two different times, the pose transformation matrix P of the left camera is calculated IL1 And the pose transformation matrix P of the right camera IR1 ;

[0010] According to the measurement data collected by the inertial measurement unit at the two different times, the pose transformation matrix P of the left camera inertial measurement unit is calculated. UL1and the pose transformation matrix P of the right camera inertial measurement unit UR1 ;

[0011] The pose transformation matrix P IL1 , the posture transformation matrix P IR1 , the posture transformation matrix P UL1 and the pose transformation matrix P UR1 Map them to the left camera coordinate system respectively to obtain the transformation matrix P of the left camera FIL , the transformation matrix P of the right camera FIR , the transformation matrix P of the left camera inertial measurement unit FUL and the transformation matrix P of the right camera inertial measurement unit FUR ;

[0012] If the transformation matrix P FIL , transformation matrix P FIR , transformation matrix P FUL and the transformation matrix P FUR are all less than the transformation threshold matrix P Thre If the corresponding setting parameters are not met, it is considered that jitter occurs between the two different moments.

[0013] Furthermore, the pose transformation matrix P of the left camera is calculated based on the image frames taken by the binocular camera at two different times. IL1 And the pose transformation matrix P of the right camera IR1 ,include:

[0014] According to the first image frames I recorded by the left camera at the two different moments L1 and the second image frame I L2 , solve the pose transformation matrix P of the left camera IL1 ;

[0015] According to the first image frames I recorded by the right camera at the two different moments R1 and the second image frame I R2 , solve the pose transformation matrix P of the right camera IR1 .

[0016] Furthermore, the position transformation matrix P of the left camera inertial measurement unit is calculated based on the measurement data collected by the inertial measurement unit at the two different times. UL1 and the pose transformation matrix P of the right camera inertial measurement unit UR1 ,include:

[0017] According to the first measurement data U recorded by the left camera inertial measurement unit at the two different times L1 and the second measurement data U L2, solve the pose transformation matrix P of the left camera inertial measurement unit UL1 ;

[0018] According to the first measurement data U recorded by the inertial measurement unit of the right camera at the two different times R1 and the second measurement data U R2 , solve the pose transformation matrix P of the right camera inertial measurement unit UR1 .

[0019] Furthermore, it also includes:

[0020] Using the set weight parameter as the weight, calculate the transformation matrix P FIL , transformation matrix P FIR , transformation matrix P FUL and the transformation matrix P FUR The weighted sum of F ;

[0021] If it is considered that jitter occurs between the two different moments, the fusion transformation matrix P is used F Perform stabilization on the output image.

[0022] Furthermore, the fusion transformation matrix P is used F Perform stabilization operations on the output image, including:

[0023] The left image of the output after calculating the stabilization operation Among them, K L is the left camera intrinsic parameter matrix;

[0024] The right image of the output after calculating the stabilization operation Among them, K R is the intrinsic parameter matrix of the right camera.

[0025] Furthermore, the transformation matrix P of the left camera is FIL =P IL1 .

[0026] Furthermore, the transformation matrix P of the right camera is FIR =P I *P IR1 .

[0027] Furthermore, the transformation matrix P of the left camera inertial measurement unit is FUL =P UL *P UL1 .

[0028] Furthermore, the transformation matrix P of the right camera inertial measurement unit is FUR =P I *P UR *PUR1 .

[0029] Furthermore, the left and right cameras of the binocular camera have a common field of view.

[0030] Beneficial Effects

[0031] Due to the adoption of the above technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the present invention installs an inertial measurement unit on each camera of the binocular camera, and solves the coordinate conversion matrix after calibrating the camera and the inertial measurement unit, solves the posture transformation matrix by using image frames and inertial measurements at different times, maps all posture transformation matrices to the left camera coordinate system and then calculates the weighted sum to obtain the fusion transformation matrix, finally judges the image frame jitter amplitude according to the fusion transformation matrix, and then decides whether to perform stabilization operation according to the jitter amplitude, thereby solving the stability problem of the binocular camera system under the requirement of precise alignment of the left and right images. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of an electronic stabilization device for a binocular camera according to an embodiment of the present invention;

[0033] Figure 2 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0035] The embodiment of the present invention relates to a stereo camera electronic stabilization method. Figure 1 The binocular camera shown in the figure includes more than two cameras, and the relative position of the two cameras is fixed and has a common field of view. An inertial measurement unit (IMU) is installed on each camera, and the relative position between the two IMUs is fixed. Each frame of image acquired by the camera and the linear velocity and angular velocity data collected by the IMU carry timestamp information, and the timestamp information of the images output by the two cameras and the linear velocity and angular velocity data output by the two IMUs is ensured to be the same through a synchronous trigger signal.

[0036] like Figure 2 As shown, the specific process is:

[0037] S1: Calibrate the binocular camera and obtain the intrinsic parameter matrix K of the left camera L , and the intrinsic parameter matrix K of the right camera R, and the external parameter transformation matrix of the right camera image to the left camera is P I .

[0038] S2: The camera and its corresponding IMU are calibrated to obtain external parameters, where the transformation matrix from the left IMU to the left camera is P UL , the transformation matrix from the right IMU to the right camera is P UR .

[0039] S3: Read and record the first frame of the left camera image I L1 and right camera image I R1 , and left IMU data U L1 and right IMU data U R1 The final left camera image output in the first frame is I LO1 =I L1 , the right camera image is I RO1 =I R1 .

[0040] S4: Read and record the second frame of left camera image I L2 and right camera image I R2 , and left IMU data U L2 and right IMU data U R2 .

[0041] S5: Pass I L1 and I L2 Calculate the pose transformation matrix P between the two frames of the left camera IL1 . And through I R1 and I R2 Calculate the pose transformation matrix P between the two frames of the right camera IR1 .

[0042] S6: Through U L1 and U L2 Calculate the pose transformation matrix P between the two frames of the left IMU UL1 . And through U R1 and U R2 Calculate the pose transformation matrix P between the two frames of the right IMU UR1 .

[0043] S7: Unify all the pose transformation matrices to the coordinates of the left camera. The transformation matrix P corresponding to the left camera FIL =P IL1 ; Transformation matrix P corresponding to the right camera FIR =P I *P IR1 ; Transformation matrix P corresponding to the left IMU FUL =P UL *P UL1 ; Transformation matrix P corresponding to the right IMUFUR =P I *P UR *P UR1 .

[0044] S8: Calculate the final fusion transformation matrix P F =k1*P FIL +k2*P FIR +k3*P FUL +k4*P FUR Among them, k1, k2, k3, k4 are specific gravity parameters, satisfying k1+k2+k3+k4=1, and all are greater than or equal to 0.

[0045] S9: If P F P FIL , P FIR , P FUL , P FUR are all less than the transformation threshold matrix P Thre The corresponding parameters in . It is considered that the jitter amplitude of this frame is small and no stabilization operation is required. The final output left camera image is I LO2 =I L2 , the right camera image is I RO2 =I R2 Otherwise, it is considered that the jitter amplitude of the frame is large and stabilization operation is required. The final output left picture is The final output on the right is

[0046] S10: Repeat steps S4 to S9 until the algorithm exits.

[0047] It is worth noting that after the external parameter calibration is completed, the relative positions of the camera and IMU cannot be changed.

Claims

1. A binocular camera stabilization method, characterized in that: The following steps are involved: Install an inertial measurement unit on each camera of the stereo camera; The binocular camera is calibrated, and the external parameter transformation matrix P of the right camera coordinate system to the left camera coordinate system is calculated based on the obtained internal parameter matrices of the left and right cameras. I ; Calibrate each camera and its inertial measurement unit of the binocular camera to obtain the transformation matrix P of the left camera inertial measurement unit coordinate system to the left camera coordinate system UL , and the transformation matrix P from the right camera inertial measurement unit coordinate system to the right camera coordinate system UR ; According to the image frames taken by the binocular camera at two different times, the pose transformation matrix P of the left camera is calculated IL1 And the pose transformation matrix P of the right camera IR1 ; According to the measurement data collected by the inertial measurement unit at the two different times, the pose transformation matrix P of the left camera inertial measurement unit is solved UL1 and the pose transformation matrix P of the right camera inertial measurement unit UR1 ; The pose transformation matrix P IL1 , the posture transformation matrix P IR1 , the posture transformation matrix P UL1 and the pose transformation matrix P UR1 Map them to the left camera coordinate system respectively to obtain the transformation matrix P of the left camera FIL , the transformation matrix P of the right camera FIR , the transformation matrix P of the left camera inertial measurement unit FUL and the transformation matrix P of the right camera inertial measurement unit FUR ; If the transformation matrix P FIL , transformation matrix P FIR , transformation matrix P FUL And the transformation matrix P FUR are all less than the transformation threshold matrix P Thre If the corresponding setting parameters are not met, it is considered that jitter occurs between the two different moments.

2. The method according to claim 1, characterized in that The pose transformation matrix P of the left camera is calculated based on the image frames taken by the binocular camera at two different times. IL1 And the pose transformation matrix P of the right camera IR1 ,include: According to the first image frames I recorded by the left camera at the two different moments L1 and the second image frame I L2 , solve the pose transformation matrix P of the left camera IL1 ; According to the first image frames I recorded by the right camera at the two different moments R1 and the second image frame I R2 , solve the pose transformation matrix P of the right camera IR1 .

3. The method according to claim 2, characterized in that The position transformation matrix P of the left camera inertial measurement unit is calculated based on the measurement data collected by the inertial measurement unit at the two different times. UL1 and the pose transformation matrix P of the right camera inertial measurement unit UR1 ,include: According to the first measurement data U recorded by the left camera inertial measurement unit at the two different times L1 and the second measurement data U L2 , solve the pose transformation matrix P of the left camera inertial measurement unit UL1 ; According to the first measurement data U recorded by the inertial measurement unit of the right camera at the two different times R1 and the second measurement data U R2 , solve the pose transformation matrix P of the right camera inertial measurement unit UR1 .

4. The method according to claim 3, characterized in that Also includes: Using the set weight parameter as the weight, calculate the transformation matrix P FIL , transformation matrix P FIR , transformation matrix P FUL and the transformation matrix P FUR The weighted sum of F ; If it is considered that jitter occurs between the two different moments, the fusion transformation matrix P is used F Perform stabilization on the output image.

5. The method according to claim 4, characterized in that The use of the fusion transformation matrix P F Perform stabilization operations on the output image, including: The left image of the output after calculating the stabilization operation Among them, K L is the left camera intrinsic parameter matrix; The right image of the output after calculating the stabilization operation Among them, K R is the intrinsic parameter matrix of the right camera.

6. The method according to claim 1, characterized in that The transformation matrix P of the left camera FIL =P IL1 .

7. The method according to claim 1, characterized in that The transformation matrix P of the right camera FIR =P I *P IR1 .

8. The method according to claim 1, characterized in that The transformation matrix P of the left camera inertial measurement unit is FUL =P UL *P UL1 .

9. The method according to claim 1, characterized in that: The transformation matrix P of the right camera inertial measurement unit is FUR =P I *P UR *P UR1 .

10. The method according to claim 1, characterized in that The left and right cameras of the binocular camera have a common field of view.

Images