A calibration method for a camera and an IMU, an electronic device, and a system

Abstract

The present application relates to the field of calibration technology, and provides a calibration method for a camera and an IMU, an electronic device and a system. Multiple calibration board images collected by the camera of the VR device are used to determine the translation information and rotation information, and the 3D B-spline and SO3-spline are initialized with the translation information and rotation information respectively, so as to solve the problem of inconsistent convergence directions when solving the external parameters due to the non-periodicity of the 6D B-spline, ensuring that the 3D B-spline and SO3-spline are continuously differentiable at any position, enabling the robotic arm where the VR device is located to move freely without being restricted by the calibration trajectory; according to the spline derivative results and the measured values of the IMU on the VR device, the target measurement error of the IMU is determined, and according to the fiducial points on multiple calibration board images, the reprojection error of the camera is determined. By solving the minimum value of the measurement error and the reprojection error, the external parameters of the camera and the IMU are accurately and stably calibrated, improving the calibration efficiency.

Description

Technical Field

[0001] This application relates to the field of calibration technology, and particularly to multi-sensor calibration. It provides a calibration method, an electronic device, and a system for a camera and an IMU. Background Art

[0002] The Simultaneous Localization and Mapping (SLAM) technology provides positioning and mapping functions for devices such as Virtual Reality (VR) and Augmented Reality (AR), and is one of the core technologies of VR, AR and other devices.

[0003] Most VR, AR and other devices implement the SLAM technology by using computer vision and inertial navigation systems (simply referred to as visual inertial navigation systems). Before using the visual inertial navigation system, it is necessary to determine the internal parameters and external parameters of the camera and the Inertial Measurement Unit (IMU) through a calibration system.

[0004] Currently, the external parameters of the camera and the IMU can be determined by differentiating the calibration trajectory. However, if the design of the calibration trajectory is unreasonable, it will affect the accuracy of the external parameter calibration and even lead to calibration failure. Therefore, most manufacturers of VR, AR and other devices will provide calibration trajectories, which impose certain constraints on the calibration process. Moreover, the calibration algorithm is related to the calibration trajectory, and the calibration algorithm requires the calibration trajectory to be continuously differentiable.

[0005] In related technologies, in order to make the calibration trajectory generated by a 6-dimensional B-Spline continuously differentiable at any position, the periodicity of the rotation angle is usually used for complementation. However, the B-Spline itself does not have periodicity. Therefore, the derivatives obtained in different periods are not consistent. In this way, when using the optimization method to solve the external parameters, the problem of inconsistent convergence directions will occur, reducing the calibration accuracy; and when the supplemented period is too large, the phenomenon of convergence error is likely to occur, ultimately leading to calibration failure. Summary of the Invention

[0006] Embodiments of this application provide a calibration method, an electronic device, and a system for a camera and an IMU, which are used to reduce the constraints of the calibration trajectory and improve the calibration accuracy and efficiency.

[0007] On the one hand, embodiments of this application provide a calibration method for a camera and an IMU. The camera and the IMU are installed on the same VR device. The method includes:

[0008] Determine the translation information and rotation information of the camera according to multiple calibration board images collected by the camera;

[0009] Initialize a 3D B-spline with the translation information and initialize an SO3 spline with the rotation information;

[0010] Derive the 3D B-spline and the SO3 spline, and determine the target measurement error of the IMU according to the derivation results and the measurement values of the IMU;

[0011] Determine the reprojection error of the camera according to the fiducial points on the multiple calibration board images;

[0012] Calibrate the extrinsic parameters between the camera and the IMU by solving the minimum values of the measurement error and the reprojection error.

[0013] On the other hand, an embodiment of the present application provides an electronic device, including a processor, a memory, and a communication interface, where the communication interface, the memory, and the processor are connected through a bus;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Obtain multiple calibration board images collected by a camera of a VR device and measurement values of an IMU of the VR device through the communication interface;

[0016] Determine the translation information and rotation information of the camera according to the multiple calibration board images;

[0017] Initialize a 3D B-spline with the translation information and initialize an SO3 spline with the rotation information;

[0018] Derive the 3D B-spline and the SO3 spline, and determine the target measurement error of the IMU according to the derivation results and the measurement values of the IMU;

[0019] Determine the reprojection error of the camera according to the fiducial points on the multiple calibration board images;

[0020] Calibrate the extrinsic parameters between the camera and the IMU by solving the minimum values of the measurement error and the reprojection error.

[0021] On the other hand, an embodiment of the present application provides a calibration system, including a calibration board, a robotic arm, a VR device, and an electronic device. The VR device includes an IMU and at least one camera. The VR device is placed at the movable end of the robotic arm and moves with the movement of the robotic arm. The fixed end of the robotic arm is connected to a robotic arm base. The camera is used to collect calibration board images from different perspectives during the movement. The accelerometer and gyroscope in the IMU are respectively used to measure acceleration and angular velocity. The electronic device executes the calibration method for the camera and IMU provided by the embodiment of the present application according to multiple calibration board images collected by the camera and the acceleration and angular velocity measured by the IMU.

[0022] On the other hand, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a computer device to execute the calibration method for the camera and IMU provided by the embodiment of the present application.

[0023] In a calibration method for a camera and an IMU, an electronic device, and a system provided by an embodiment of the present application, multiple calibration board images collected by a camera on a VR device are used to determine the translation information and rotation information of the camera. The 3D B-spline and SO3-spline are respectively initialized with the translation information and rotation information, and the 3D B-spline and SO3-spline are respectively differentiated. Compared with the 6D B-spline, the SO3-spline takes into account the periodicity of rotation, ensuring that both the 3D B-spline and SO3-spline are continuously differentiable, enabling the carrier where the VR is located to move freely without being restricted by the calibration trajectory. Further, according to the differentiation results and the measured values of the IMU on the VR device, the target measurement error of the IMU is determined, and the reprojection error of the camera is determined based on the fiducial points on multiple calibration board images. By solving the minimum value of the measurement error and the reprojection error, the external parameters of the camera and IMU are accurately and stably calibrated. Compared with the SE3-spline, the solution speed is fast, improving the calibration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1A It is a surface structure diagram of the VR device provided by an embodiment of the present application;

[0026] Figure 1B It is an internal structure diagram of the VR device provided by an embodiment of the present application;

[0027] Figure 2 Flowchart of the calibration method using 6D B-spline provided by an embodiment of the present application;

[0028] Figure 3A System architecture diagram of the calibration system provided by an embodiment of the present application;

[0029] Figure 3B Partial schematic diagram of the positional relationship between the VR device and the robotic arm provided by an embodiment of the present application;

[0030] Figure 3C Side view of the movement process of the robotic arm provided by an embodiment of the present application;

[0031] Figure 3D Calibration framework diagram of the camera and the IMU provided by an embodiment of the present application;

[0032] Figure 4 Flowchart of the calibration method of the camera and the IMU provided by an embodiment of the present application;

[0033] Figure 5 Initialization schematic diagram of the 3D B-spline provided by an embodiment of the present application;

[0034] Figure 6 Flowchart of the initialization method of the SO3 spline provided by an embodiment of the present application;

[0035] Figure 7 Initialization schematic diagram of the SO3 spline provided by an embodiment of the present application;

[0036] Figure 8 Flowchart of the method for determining the target measurement error of the IMU provided by an embodiment of the present application;

[0037] Figure 9 Flowchart of the method for determining the reprojection error of the camera provided by an embodiment of the present application;

[0038] Figure 10 Structural diagram of the electronic device provided by an embodiment of the present application;

[0039] Figure 11 Structural diagram of the calibration system provided by an embodiment of the present application. Detailed implementation manners

[0040] Most devices such as VR and AR adopt computer vision and inertial navigation systems (abbreviated as visual inertial navigation systems) to implement the SLAM technology. Before using the visual inertial navigation system, it is necessary to calibrate the internal parameters of the two sensors, namely the camera and the IMU, on the VR device, as well as their external parameters. Among them, the calibration of the internal parameters is quite mature and is not the focus of the embodiments of the present application. The embodiments of the present application focus on the calibration of the external parameters of the camera and the IMU.

[0041] Taking a VR device as an example, refer to Figure 1A , which shows the surface structure diagram of the VR device. A camera is installed at each of the four corners of the VR device, namely the upper, lower, left, and right corners. The IMU is located inside the VR device and cannot be seen from the outer surface of the VR device. Refer to Figure 1B for the internal structure of the VR device shown. In Figure 1A and Figure 1B , the positions of the four cameras and the IMU on the VR device are circled respectively.

[0042] Generally, when jointly calibrating the cameras and the IMU on the VR device, the VR device can be installed on a robotic arm, and the movement of the robotic arm drives the movement of the cameras and the IMU. The movement of the robotic arm should ensure sufficient excitation of the sensors. For the camera sensor, sufficient excitation means that the camera's observation of the calibration board needs to cover the entire image area. For the IMU sensor, sufficient excitation means that the acceleration and angular velocity of the robotic arm movement should reach a certain magnitude. If the calibration trajectory of the robotic arm is designed unreasonably, it will lead to inaccurate calibration or even failure. Therefore, most manufacturers of VR, AR, and other devices will provide calibration trajectories, which impose certain constraints on the calibration process.

[0043] In addition, during the joint calibration process of the cameras and the IMU, pose information can be obtained from the calibration board images captured by the cameras, and acceleration information and angular velocity information can be obtained from the data measured by the accelerometers and gyroscopes of the IMU. To establish the interconnections among the calibration board, cameras, and IMU, B-Spline is usually used to model the pose information of the cameras to obtain the calibration trajectory, and the relationship between the IMU and acceleration / angular velocity is obtained by differentiating the calibration trajectory, thereby completing the calibration of the cameras and the IMU.

[0044] The calibration trajectory is related to the calibration algorithm. Currently, 6D B-Spline is generally used to model the pose information. Among them, these 6 dimensions respectively represent the translation (displacement) vector and rotation matrix of the body coordinate system, that is, pose = [px, py, pz, rx, ry, rz] T . Since B-Spline constructs the time series of pose, the integration and differentiation of the 6D vector with respect to time are calculated independently. By combining the calculation results, the translation vector of the body coordinate system is obtained from the first three dimensions, and the rotation matrix of the body coordinate system is obtained from the last three dimensions. The formulas are as follows:

[0045] p(px, py, pz, t) = g0(px(t), py(t), pz(t)) Formula 1

[0046] R(rx, ry, rz, t) = g1(rx(t), ry(t), rz(t)) Formula 2

[0047] Among them, p represents the translation vector, R represents the rotation matrix, g0 and g1 are the mapping equations of the 6D B-Spline for the translation vector and the rotation matrix respectively, t represents time, (px, py, pz) represents the three-axis coordinates, and (rx, ry, rz) represents the three-axis attitude angles.

[0048] The process of calibrating the camera and the IMU using the 6D B-Spline is shown in Figure 2 , and mainly includes the following steps:

[0049] S201: Initialize the time offset parameter (TimeShift) between the camera and the IMU.

[0050] S202: Initialize the pose spline of the IMU using the 6D B-Spline.

[0051] S203: Initialize the bias spline of the IMU using the 3D B-Spline to reduce the measurement error of the IMU.

[0052] S204: Construct a residual optimization equation based on the reprojection error of the camera and the measurement errors of the accelerometer and gyroscope of the IMU. Among them, the pose (including translation and rotation), angular velocity, and acceleration of the IMU are obtained according to the 6D B-Spline.

[0053] During the Figure 2 calibration process shown, the calibration algorithm requires that the calibration trajectory be continuously differentiable at any position. If the problem of continuous differentiability is not solved, the robotic arm can only move within one cycle of 360°, and the obtained calibration trajectory is only within one motion cycle, which will affect the design of the calibration trajectory and the user experience of the device, especially when calibrating using the box of a 3-axis robotic arm.

[0054] Considering the particularity of the rotation angle in the pose: any rotation in space can be represented as a rotation around a specific axis by a specific angle, and the rotation angle is a periodic function with a period of 2π. In order to make the B-Spline continuously differentiable at any position, the periodicity of the rotation angle is usually used for complementation. However, the B-Spline itself does not have periodicity. Therefore, when taking the derivative of R in different periods, the obtained derivatives are not consistent.

[0055] For example, assume that R(rx1, ry1, rz1, t) = R(rx2, ry2, rz2, t). Due to periodicity, two identical trajectories have different derivative results, and the formula is as follows:

[0056]

[0057] It can be seen that when differentiating the calibrated trajectory obtained from the 6D B-Spline, it is difficult to consider the particularity of the rotation angle. In this way, when using the optimization method to solve the external parameters, the problem of inconsistent convergence directions will occur, reducing the calibration accuracy. Moreover, when the supplementary period is too large, the phenomenon of convergence errors is likely to occur, ultimately leading to calibration failure, and this situation occurs more frequently in the box calibration system.

[0058] In related technologies, to solve the problems existing in the 6D B-Spline, the SE3 spline algorithm proposed for poses is used to solve the periodicity problem. However, in the process of solving the SE3 spline, the involved box-minus operation cannot obtain the explanatory solution of its related derivative:

[0059]

[0060] Among them, se3 represents the Lie algebra corresponding to the SE3 spline, SE3 = exp(se3), and a and b are distinguishing identifiers representing two different se3s.

[0061] Since the box-minus operation cannot obtain the explanatory solution of its related derivative, it greatly affects the solution speed and the calibration efficiency.

[0062] In view of this, the embodiments of the present application provide a calibration method and system for a camera and an IMU, which jointly calibrate the camera and the IMU using 3D B-Spline and SO3 spline. This method considers the periodicity of rotation and can fully solve the above problems, enabling the robotic arm to move freely during the calibration process, removing the restrictions on the calibration trajectory, so that the calibration carriers (camera and IMU) can be fully excited, outputting accurate and stable external parameters, providing accurate external parameter configurations for the camera and the IMU for the SLAM system, and supporting the stable operation of the SLAM system.

[0063] See Figure 3A , which is the architecture diagram of the calibration system for the camera and the IMU provided by the embodiments of the present application. The system includes a calibration board, a robotic arm, a VR device with at least one camera and an IMU, and a host. Considering that the orientations of multiple cameras on the VR device are different, if a planar calibration board is used, some cameras may not be able to capture the calibration board during the movement process, or the captured area of the calibration board is small. Therefore, a non-planar calibration board with a fold angle is used. The fixed end of the robotic arm is connected to the base, and the movable end is equipped with a VR device. The local structure of the VR device and the robotic arm is as Figure 3B shown. The movement of the robotic arm is controlled by a remote control, thereby controlling the movement of the camera and the IMU. The side view of the movement of the robotic arm is as Figure 3C shown.

[0064] During the calibration process, the camera takes images of the calibration plate at different viewing angles and sends them to the host. The motion data measured by the accelerometer and gyroscope of the IMU is also sent to the host. The host uses the 3D B-Spline and SO3 spline joint calibration method to complete the calibration of the camera and IMU based on the calibration plate images sent by the camera and the motion data sent by the IMU. This method ensures the continuity of the calibration trajectory derivation, and the adjustment of the calibration trajectory is not limited by the rotation periodicity, which improves the calibration accuracy.

[0065] It should be noted that Figure 3A The host and VR device are only examples and are not intended to be limiting descriptions of this application. Figure 3A In addition to the laptop computer, it can also be any electronic device with computing power, such as a desktop computer. The VR device can be a VR helmet and VR glasses.

[0066] In some embodiments, if the VR device has an independent processor and the computing performance of the processor is good, the VR device can also complete the calibration of the camera and the IMU based on the calibration plate image sent by the camera and the motion data sent by the IMU.

[0067] It should be noted that before calibrating the external parameters of the camera and IMU, the internal parameters of the two have been calibrated in advance, and when the VR device has multiple cameras, the calibration method of each camera is the same. Therefore, the calibration process is described below using one camera as an example.

[0068] based on Figure 3A The calibration system shown is for the camera and IMU in the VR device. Figure 3D The overall framework diagram of the camera and IMU calibration provided by the embodiment of the present application is shown as an example. The framework diagram mainly includes 4 parts, the first part S301 is: initializing the time offset parameter (TimeShift) between the camera and the IMU; the second part S302 is: initializing the pose spline of the IMU and the camera, wherein the 3D B-spline is initialized with the translation information, and the SO3 spline is initialized with the rotation information; the third part S303 is: using the 3D B-Spline to initialize the deviation spline of the IMU to reduce the measurement error of the IMU; the fourth part S304 is: determining the reprojection error of the camera and the measurement error of the IMU, and obtaining the joint calibration optimization equation to solve the external parameters of the camera and the IMU, wherein the translation of the IMU and the camera, and the acceleration of the IMU are obtained according to the 3D B-spline, and the rotation of the IMU and the camera, and the angular velocity of the IMU are obtained according to the SO3 spline.

[0069] based on Figure 3A The calibration system shown and Figure 3D The overall calibration framework shown, Figure 4An exemplary flowchart of the calibration method for the camera and IMU according to the embodiments of the present application is shown. This method is executed by an electronic device with computing capabilities and mainly includes the following steps:

[0070] S401: Determine the translation information and rotation information of the camera based on multiple calibration board images collected by the camera on the VR device.

[0071] In an alternative embodiment, the VR device is fixed to the moving end of the robotic arm and moves with the movement of the robotic arm. In this way, during the movement, the camera on the VR device can collect multiple calibration board images. The Zhang Zhengyou calibration method or the MATLAB toolbox is used to extract the fiducial points in each calibration board image, and based on the extracted fiducial points, the translation information and rotation information of the camera are determined.

[0072] S402: Initialize the 3D B-spline with the translation information and initialize the SO3 spline with the rotation information.

[0073] As can be seen from the description of the 6D B-Spline above, the translation vector of the body coordinate system can be obtained from the first 3 dimensions, and the translation vector does not have periodicity, nor does the B-spline itself. Therefore, the translation information of the camera can be modeled to obtain a 3D B-spline containing the translation information.

[0074] For the schematic diagram of the initialization of the 3D B-spline, see Figure 5 , for the translation information at n moments, a translation line segment is determined for every two adjacent translation information, and curve fitting is performed on the n - 1 translation line segments to obtain a 3D B-spline containing the translation information, where n is an integer greater than or equal to 2.

[0075] According to the definition of the SO3 spline, the spline value at time t is determined by 4 control points. Assume t ∈ [t i , t i+1 , then the times corresponding to these 4 control points are [t i-1 , t i , t i+1 , t i+2 . Define the normalized time function s(t) = (t - t i ) / (t i+1 - t i ), for the given time s(t) within the time range [s i , s i+1 ), define the piecewise function f(t) = s(t) - s i , at this time, the basis matrix of the SO3 spline can be obtained as:

[0076]

[0077] Among them, C is the coefficient matrix, specifically:

[0078]

[0079] At this time, at time t (the normalized time is s, and the times corresponding to the 4 control points are [t i-1 , t i , t i+1 , t i+2 ), the rotation matrix is:

[0080] R ω，s = R ω，i-1 A0A1A2 Formula 7

[0081] where A j = exp(B j+1 (u)Ω i-1+j，i+j ), B j+1 (f) represents the (j + 1)-th row in the basis matrix B(f) of the SO3 spline, Ω i-1，i is the spline value increment between two times (t i-1 , t i ), A j is the intermediate matrix, and j = 0, 1, 2.

[0082] In the embodiments of the present application, the spline values of the SO3 spline are the rotation information at each time. For the process of initializing the SO3 spline with the rotation information of the camera, see Figure 6 , which mainly includes the following steps:

[0083] S4021: Obtain the rotation information at 4 times.

[0084] As Figure 7 shown, assume that these 4 times are [t i-1 , t i , t i+1 , t i+2 , and the obtained rotation information is [R ω，i-1 , R ω，i , R ω，i+1 , R ω，i+2 .

[0085] S4022: Determine the piecewise function at 4 times, and determine 3 rotation line segments according to the piecewise function.

[0086] According to the definition of the SO3 spline, the normalized time function is s(t) = (t - t i ) / (t i+1 - t i ), and the piecewise function is u(t) = s(t) - s i, according to the segmentation result, combine every two adjacent rotation information among the information at 4 moments into a rotation line segment, and obtain 3 rotation line segments. See Figure 7 .

[0087] S4023: Perform curve fitting on the 3 rotation line segments to obtain a SO3 spline.

[0088] Perform curve fitting on the 3 rotation line segments so that the fitted curve passes through the rotation information at 4 moments, thereby obtaining a SO3 spline curve containing rotation information.

[0089] In the embodiments of the present application, considering the particularity of rotation, two splines are respectively fitted with translation information and rotation information, thereby solving the problem of inconsistent convergence directions when using the optimization method to solve the external parameters due to the non-periodicity of the B-spline, and improving the accuracy of calibration.

[0090] It should be noted that since the camera and the IMU are located on the same VR device, therefore, the translation information and rotation information of the camera can also be used as the translation information and rotation information of the IMU. In this way, the obtained 3D B-spline and SO3 spline can also represent the translation and rotation of the IMU in the inertial coordinate system (body coordinate system) respectively.

[0091] S403: Differentiate the 3D B-spline and the SO3 spline, and determine the target measurement error of the IMU according to the differentiation result and the measurement value of the IMU on the VR device.

[0092] According to the definitions of angular velocity and linear acceleration, it can be obtained that:

[0093]

[0094]

[0095] Among them, ω represents angular velocity, a represents linear acceleration, R ω，s represents the spline interpolation corresponding to the normalized time s in the SO3 spline (i.e., rotation information), p ω，s represents the spline interpolation corresponding to the normalized time s in the 3D B-spline (i.e., translation information), ∧ represents vector diagonal matrixization, v represents the vectorization of the skew-symmetric matrix, and T represents the transpose of the matrix.

[0096] Among them, the operation relationships of vector diagonal matrixization and the vectorization of the skew-symmetric matrix are as follows:

[0097]

[0098]

[0099] When executing S403, according to the definitions of angular velocity and linear acceleration, it can be known that by taking the derivatives of the 3D B-spline and SO3-spline, the calculated values of angular velocity and linear acceleration can be obtained. Then, combined with the measured values of the accelerometer and gyroscope in the IMU of the VR device, the target measurement error of the IMU can be determined. For the specific process, refer to Figure 8 , which mainly includes the following steps:

[0100] S4031: Determine the calculated value of acceleration according to the second derivative of the 3D B-spline.

[0101] In the embodiments of the present application, the translation information is the displacement information. The displacement is a function of time. Taking the first derivative of the displacement with respect to time is the velocity, and taking the derivative of the velocity again is the acceleration. Therefore, in S4031, assuming that the 3D B-spline containing the translation information is p ω，s , take the second derivative of p ω，s , and according to the definition of acceleration in Formula 9, the calculated value of acceleration can be obtained

[0102] S4032: Determine the acceleration measurement error according to the calculated value of acceleration, gravitational acceleration, the acceleration measurement value of the accelerometer in the IMU, and the acceleration measurement deviation.

[0103] In the embodiments of the present application, the IMU includes an accelerometer and a gyroscope for measuring acceleration and angular velocity. There are measurement deviations in the accelerometer and gyroscope themselves. Therefore, when calculating the acceleration measurement error of the IMU in S4032, the measurement deviations of the accelerometer and gyroscope should be considered to improve the accuracy of the calculation of the accelerometer measurement error, and further improve the accuracy of calibration.

[0104] In an optional implementation, first, initialize the bias spline of the IMU with the 3D B-spline. Then, obtain the measurement deviation of the accelerometer (denoted as the acceleration measurement deviation, denoted as ba) and the measurement deviation of the angular velocity (denoted as the angular velocity measurement deviation, denoted as bg) from the bias spline of the IMU.

[0105] In S4032, the acceleration measurement value can be directly read out through the accelerometer in the IMU. Combining the calculated value of acceleration, gravitational acceleration, and the acceleration measurement deviation, the acceleration measurement error of the IMU can be obtained:

[0106]

[0107] S4033: Determine the calculated value of angular velocity according to the first derivative of the SO3-spline.

[0108] The rotation angle is the product of the angular velocity and time. Therefore, in S4033, for the SO3-spline R containing the rotation information ω，sFind the first derivative. According to the definition of angular velocity in Formula 8, the calculated value of angular velocity can be obtained.

[0109] S4034: Determine the angular velocity measurement error based on the calculated value of angular velocity, the measured value of angular velocity of the gyroscope in the IMU, and the angular velocity measurement deviation.

[0110] In S4034, the measured value of angular velocity can be directly read out by the gyroscope in the IMU. Combining the calculated value of angular velocity and the angular velocity measurement deviation, the angular velocity measurement error of the IMU can be obtained:

[0111]

[0112] S4035: Determine the target measurement error of the IMU based on the acceleration measurement error and the angular velocity measurement error.

[0113] Among them, the target measurement error of the IMU is:

[0114]

[0115] Among them, R ω，s represents the rotation information corresponding to the normalized time s in the SO3 spline, p ω，s represents the translation information corresponding to the normalized time s in the 3D B-spline, is the second derivative of the 3D B-spline at the normalized time s, is the first derivative of the SO3 spline at the normalized time s, g represents the gravitational acceleration, acc represents the measured value of acceleration, ba represents the acceleration measurement deviation, gyro represents the measured value of angular velocity, bg represents the acceleration measurement deviation, and V represents the vectorization of the skew-symmetric matrix.

[0116] S404: Determine the reprojection error of the camera based on the fiducial points on multiple calibration board images.

[0117] There are corresponding pixel points of the fiducial points on the calibration board in the three-dimensional space in the two-dimensional calibration board image. Therefore, according to the projection relationship between the three-dimensional space and the two-dimensional image, the reprojection error of the camera can be determined. The specific process is shown in Figure 9 , which mainly includes the following steps:

[0118] S4041: Use the external parameters of the camera and the IMU to project the fiducial points on the calibration board from the three-dimensional space to the two-dimensional calibration board image.

[0119] In S4041, according to the projection relationship between the three-dimensional space and the two-dimensional image determined by the external parameters of the camera and the external parameters of the IMU, the fiducial points on the calibration board are projected from the three-dimensional space to the two-dimensional calibration board image. The projection relationship is as follows:

[0120]

[0121] Among them, π(·) represents the projection function of the camera, represents the extrinsic parameters of the camera, respectively represent the extrinsic parameters of the IMU, represents the rotation information corresponding to the normalized time s in the SO3 spline, p ω，s represents the translation information in the 3D B-spline corresponding to the normalized time s, q ω represents the three-dimensional coordinates of a fiducial point on the calibration board, and [u′, v′] represents the projection coordinates of the fiducial point on the corresponding calibration board image.

[0122] S4042: Determine the reprojection error of the camera according to the projection coordinates of each fiducial point and the two-dimensional coordinates of the pixel points corresponding to each fiducial point on the corresponding calibration board image.

[0123] Among them, the reprojection error of the camera is:

[0124]

[0125] Among them, [u, v] represents the two-dimensional coordinates of the pixel points corresponding to each fiducial point on the corresponding calibration board image.

[0126] S405: Calibrate the extrinsic parameters of the camera and the IMU by solving the minimum value of the measurement error and the reprojection error.

[0127] In S405, the measurement error e_imu of the IMU and the reprojection error e_camera of the camera constitute the optimization equation for the joint calibration of the camera and the IMU. By solving the extrinsic parameters of the camera and the IMU when the measurement error and the reprojection error are minimized, the optimal extrinsic parameters are obtained, and the joint calibration of the camera and the IMU is completed.

[0128] As can be seen from the above calibration method of the camera and the IMU provided by the embodiments of the present application, the translation of the IMU and the camera, and the acceleration of the IMU are obtained according to the 3D B-spline, and the rotation of the IMU and the camera, and the angular velocity of the IMU are obtained according to the SO3 spline.

[0129] In a calibration method for a camera and an IMU provided by an embodiment of the present application, multiple calibration board images collected by a camera on a VR device are used to determine the translation information and rotation information of the camera. The 3D B-spline and SO3-spline are initialized with the translation information and rotation information respectively, thereby solving the problem of inconsistent convergence directions when using an optimization method to solve the external parameters due to the non-periodicity of the B-spline. When taking the derivatives of the 3D B-spline and SO3-spline, compared with the 6D B-spline, the SO3-spline takes into account the periodicity of rotation, ensuring that the 3D B-spline and SO3-spline are continuously differentiable at any position, enabling the carrier where the VR is located (i.e., the robotic arm) to move freely without being restricted by the calibration trajectory; further, according to the second derivative of the 3D B-spline and the measured values of the accelerometer in the IMU, the acceleration measurement error of the IMU is determined, and according to the first derivative of the SO3-spline and the measured values of the gyroscope in the IMU, the angular velocity measurement error of the IMU is determined. Then, based on the acceleration measurement error and angular velocity measurement error of the IMU, the target measurement error of the IMU is obtained, and by projecting the fiducial points on the calibration board in three-dimensional space onto the two-dimensional calibration board image, the reprojection error of the camera is determined. By solving the external parameters when the measurement error of the IMU and the reprojection error of the camera are minimized, the external parameters of the camera and the IMU are accurately and stably calibrated. Compared with the SE3-spline, the solution speed is fast, improving the calibration efficiency.

[0130] Based on the same technical concept, an embodiment of the present application provides an electronic device, which can be a notebook computer, a desktop computer, etc. with computing capabilities, capable of implementing the steps of the calibration method for the camera and the IMU in the above embodiment and achieving the same technical effects.

[0131] See Figure 10 , the electronic device includes a processor 1001, a memory 1002, and a communication interface 1003. The communication interface 1003, the memory 1002, and the processor 1001 are connected through a bus 1004;

[0132] The memory 1002 stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0133] Through the communication interface 1003, obtain multiple calibration board images collected by the camera of the VR device and the measured values of the IMU of the VR device;

[0134] According to the multiple calibration board images, determine the translation information and rotation information of the camera;

[0135] Initialize the 3D B-spline with the translation information and initialize the SO3-spline with the rotation information;

[0136] Derive the 3D B-spline and the SO3-spline, and determine the target measurement error of the IMU based on the derivation results and the measurement values of the IMU;

[0137] Determine the reprojection error of the camera based on the fiducial points on the multiple calibration board images;

[0138] Calibrate the extrinsic parameters between the camera and the IMU by solving the minimum values of the measurement error and the reprojection error.

[0139] Optionally, the processor 1001 initializes the SO3-spline with the rotation information. The specific operation is as follows:

[0140] Obtain the rotation information at four moments;

[0141] Determine the piecewise functions at four moments, and determine three rotation line segments based on the piecewise functions;

[0142] Perform curve fitting on the three rotation line segments to obtain the SO3-spline.

[0143] Optionally, the processor 1001 derives the 3D B-spline and the SO3-spline, and determines the target measurement error of the IMU based on the derivation results and the measurement values of the IMU. The specific operation is as follows:

[0144] Determine the acceleration calculation value based on the second derivative of the 3D B-spline;

[0145] Determine the acceleration measurement error based on the acceleration calculation value, the gravitational acceleration, the acceleration measurement value of the accelerometer in the IMU, and the acceleration measurement deviation;

[0146] Determine the angular velocity calculation value based on the first derivative of the SO3-spline;

[0147] Determine the angular velocity measurement error based on the angular velocity calculation value, the angular velocity measurement value of the gyroscope in the IMU, and the angular velocity measurement deviation;

[0148] Determine the target measurement error of the IMU based on the acceleration measurement error and the angular velocity measurement error.

[0149] Optionally, the formula for the target measurement error of the IMU is:

[0150]

[0151] where R ω，s represents the rotation information corresponding to the normalized time s in the SO3-spline, and p ω，s represents the translation information corresponding to the normalized time s in the 3D B-spline. Denotes the second derivative of the 3D B-spline at the normalized time s. Denotes the first derivative of the SO3-spline at the normalized time s, g denotes the acceleration due to gravity, acc denotes the acceleration measurement value, ba denotes the acceleration measurement bias, gyro denotes the angular velocity measurement value, bg denotes the angular velocity measurement bias, T denotes the transpose of a matrix, and v denotes the vectorization of an anti-symmetric matrix.

[0152] Optionally, the acceleration measurement bias and the angular velocity measurement bias are obtained from the bias splines of the IMU, and the bias splines are initialized by the 3D B-spline.

[0153] Optionally, the processor 1001 determines the reprojection error of the camera according to the fiducial points on the multiple calibration board images. The specific operation is as follows:

[0154] Using the extrinsic parameters of the camera and the IMU, project the fiducial points on the calibration board from the three-dimensional space onto the two-dimensional calibration board image;

[0155] According to the projection coordinates of the fiducial points and the two-dimensional coordinates of the pixel points corresponding to the fiducial points on the corresponding calibration board image, determine the reprojection error of the camera.

[0156] Optionally, the formula for the reprojection error of the camera is:

[0157]

[0158] where π(·) represents the projection function of the camera, and respectively represent the extrinsic parameters of the camera and the IMU, represents the rotation information of the normalized time s in the SO3-spline, p ω，s represents the translation information of the normalized time s in the 3D B-spline, q ω represents the three-dimensional coordinates of a fiducial point on the calibration board, and [u, v] represents the two-dimensional coordinates of the pixel point corresponding to the fiducial point on the corresponding calibration board image.

[0159] It should be noted that Figure 9 is only an example, showing the necessary hardware for the electronic device to execute the steps of the calibration method of the camera and the IMU provided by the embodiments of the present application. If not shown, the electronic device may further include a display screen, a mouse, a keyboard, etc.

[0160] Embodiments of the present application Figure 10The processor involved may be a Central Processing Unit (CPU), a general-purpose processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application-specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0161] Based on the same technical concept, an embodiment of this application provides a calibration system, which can implement the calibration method steps of the camera and IMU of the VR device in the above embodiment and achieve the same technical effects.

[0162] See Figure 11 , the calibration system includes a calibration board 1101, a robotic arm 1102, a VR device 1103, and an electronic device 1104. Among them, the VR device 1103 includes an IMU 1103_1 and at least one camera 1103_2. The VR device 1103 is placed at the movable end of the robotic arm and moves with the movement of the robotic arm. The fixed end of the robotic arm is connected to the robotic arm base. The camera 1103_2 is used to collect calibration board images from different perspectives during the movement. The accelerometer 1103_11 and gyroscope 1103_12 in the IMU 1103_1 are respectively used to measure acceleration and angular velocity. The electronic device 1104 realizes the steps of the calibration method of the camera and IMU shown in the above embodiment according to the multiple calibration board images collected by the camera 1103_2 and the acceleration and angular velocity measured by the IMU 1103_1. Figure 4 The steps of the calibration method of the camera and IMU shown.

[0163] Those skilled in the art should understand that the embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a means for implementing the functions specified in one or more flows of the flowchart and / or one or more blocks of the block diagram.

[0165] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction means that implements the functions specified in one or more flows of the flowchart and / or one or more blocks of the block diagram.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are executed on the computer or other programmable device to generate a computer-implemented process, thereby providing steps for implementing the functions specified in one or more flows of the flowchart and / or one or more blocks of the block diagram by the instructions executed on the computer or other programmable device.

[0167] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A calibration method for a camera and an IMU, characterized in that The camera and the IMU are installed on the same VR device, and the method includes: Determine the translation information and rotation information of the camera according to multiple calibration board images collected by the camera; Initialize a 3D B-spline with the translation information and initialize an SO3-spline with the rotation information; Derive the 3D B-spline and the SO3-spline, and determine the target measurement error of the IMU according to the derivation results and the measurement values of the IMU; Determine the reprojection error of the camera according to the fiducial points on the multiple calibration board images; Calibrate the external parameters of the camera and the IMU by solving the minimum values of the measurement error and the reprojection error.

2. The method according to claim 1, characterized in that The initializing the SO3-spline with the rotation information includes: Obtain the rotation information at four moments; Determine the piecewise function at four moments, and determine three rotation line segments according to the piecewise function; Perform curve fitting on the three rotation line segments to obtain an SO3-spline.

3. The method according to claim 1, characterized in that, The deriving the 3D B-spline and the SO3-spline, and determining the target measurement error of the IMU according to the derivation results and the measurement values of the IMU includes: Determine the acceleration calculation value according to the second derivative of the 3D B-spline; Determine the acceleration measurement error according to the acceleration calculation value, the gravitational acceleration, the acceleration measurement value of the accelerometer in the IMU, and the acceleration measurement deviation; Determine the angular velocity calculation value according to the first derivative of the SO3-spline; Determine the angular velocity measurement error according to the angular velocity calculation value, the angular velocity measurement value of the gyroscope in the IMU, and the angular velocity measurement deviation; Determine the target measurement error of the IMU according to the acceleration measurement error and the angular velocity measurement error.

4. The method according to claim 3, wherein The formula for the target measurement error of the IMU is: where, R ω,s represents the rotation information corresponding to the normalized time s in the SO3 spline, p ω,s represents the translation information corresponding to the normalized time s in the 3D B-spline, represents the second derivative of the 3D B-spline at the normalized time s, represents the first derivative of the SO3 spline at the normalized time s, g represents the gravitational acceleration, acc represents the acceleration measurement value, ba represents the acceleration measurement deviation, gyro represents the angular velocity measurement value, bg represents the acceleration measurement deviation, T represents the transpose of the matrix, and V represents the vectorization of the skew-symmetric matrix.

5. The method according to claim 3 or 4, characterized in that, The acceleration measurement deviation and the angular velocity measurement deviation are obtained from the deviation spline of the IMU, and the deviation spline is initialized by the 3D B-spline.

6. The method according to claim 1, wherein The determining the reprojection error of the camera according to the fiducial points on the multiple calibration board images includes: Use the external parameters of the camera and the IMU to project the fiducial points on the calibration board from three-dimensional space onto a two-dimensional calibration board image; Determine the reprojection error of the camera according to the projection coordinates of the fiducial points and the two-dimensional coordinates of the pixel points corresponding to the fiducial points on the corresponding calibration board image.

7. The method according to claim 6, wherein The formula for the reprojection error of the camera is: wherein, not(·) represents the projection function of the camera, and respectively represent the extrinsic parameters between the camera and the IMU, represents the rotation information of the normalized time s in the SO3 spline, p ω,s represents the translation information of the normalized time s in the 3D B-spline, q ω represents the three-dimensional coordinates of a fiducial point on the calibration board, and [u, v] represents the two-dimensional coordinates of the pixel point corresponding to the fiducial point on the corresponding calibration board image.

8. An electronic device, characterized in that, It includes a processor, a memory, and a communication interface, and the communication interface, the memory, and the processor are connected by a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Through the communication interface, obtain multiple calibration board images collected by the camera of the VR device and the measurement values of the IMU of the VR device; Determine the translation information and rotation information of the camera according to the multiple calibration board images; Initialize a 3D B-spline with the translation information and initialize an SO3-spline with the rotation information; Derive the 3D B-spline and the SO3-spline, and determine the target measurement error of the IMU according to the derivation result and the measurement value of the IMU; Determine the reprojection error of the camera according to the fiducial points on the multiple calibration board images; Calibrate the extrinsic parameters between the camera and the IMU by solving the minimum value of the measurement error and the reprojection error; 9. The electronic device according to claim 8, wherein The processor initializes the SO3-spline with the rotation information, and the specific operation is as follows: Obtain the rotation information at four moments; Determine the piecewise functions at four moments, and determine three rotation line segments according to the piecewise functions; Perform curve fitting on the three rotation line segments to obtain the SO3-spline; 10. A calibration system, characterized in that, It includes a calibration board, a robotic arm, a VR device, and an electronic device. The VR device includes an IMU and at least one camera. The VR device is placed at the movable end of the robotic arm and moves with the movement of the robotic arm. The fixed end of the robotic arm is connected to the robotic arm base. The camera is used to collect calibration board images from different perspectives during the movement. The accelerometer and gyroscope in the IMU are respectively used to measure acceleration and angular velocity. The electronic device executes the method according to any one of claims 1-7 based on the multiple calibration board images collected by the camera and the acceleration and angular velocity measured by the IMU.

Images