Camera-IMU joint calibration method applied to unmanned automation equipment

Abstract

The invention discloses a Camera-IMU joint calibration method applied to unmanned automation equipment, and belongs to the technical field of multi-sensor calibration. According to the method, a three-dimensional calibration structure with a plurality of non-coplanar planes is constructed, angular point information is extracted to calculate the initial pose of a camera, and multi-frame joint optimization is performed based on a beam adjustment method; and constructing a hand-eye calibration model through the IMU trajectory after interpolation alignment, solving a rotation parameter by adopting quaternion, estimating a translation parameter in combination with a linear equation, and finally obtaining a high-precision external parameter through optimization of a nonlinear least square method. The method has the advantages of being high in calibration precision, high in degradation resistance, wide in adaptability, good in deployability and the like, and is suitable for high-precision calibration of cameras and IMU sensors of various platforms such as automatic driving, mobile robots, AR / VR and the like.

Description

Technical Field

[0001] This invention belongs to the field of sensor calibration technology, specifically relating to a joint calibration method for Camera-IMU applied to unmanned automated equipment. Background Technology

[0002] With the rapid development of autonomous driving, mobile robots, and augmented reality / virtual reality (AR / VR) technologies, the demand for accurate calibration in multi-sensor fusion systems is becoming increasingly prominent. Among these, the fusion calibration of the camera and inertial measurement unit (IMU) is a core technical aspect, directly determining the positioning accuracy and stability of the entire system. In practical applications, visual sensors provide rich environmental features and absolute pose references, while the IMU, with its high-frequency characteristics (typically 200-1000Hz), achieves accurate short-term motion prediction. The complementary advantages of these two technologies enable the fusion system to maintain robust positioning performance in complex environments. However, to achieve this complementary advantage, the problem of accurate spatiotemporal alignment between the camera and IMU must first be solved, which places extremely high demands on sensor calibration technology.

[0003] While current mainstream calibration methods have made some progress, they still have significant shortcomings in terms of calibration accuracy, efficiency, and applicability, urgently requiring innovative solutions. Currently, the camera-IMU calibration methods commonly used in industry and academia are mainly based on two-dimensional planar calibration boards, such as the classic checkerboard or AprilTag calibration boards. These methods acquire images of the calibration board at different poses, combine them with IMU motion data, and use optimization algorithms to solve for the extrinsic parameter matrix and time offset between sensors. Typical open-source implementations include calibration toolchains such as Kalibr and VINS-Fusion. Although these methods meet basic requirements to some extent, their inherent technical defects severely restrict further improvements in system performance. First, from the perspective of calibration principles, the single-plane constraint of the two-dimensional calibration board results in an unobservable dimension in the system. When the angle between the camera optical axis and the calibration board normal vector is less than a critical value (usually 5°-10°), it leads to severe calibration degradation problems. Specifically, the errors manifest in several ways: in the calibration of translation parameters along the depth direction (Z-axis), the error can be amplified to over 300% of the actual value (data from MIT Draper Lab in 2018); regarding rotation parameters, the calibration error of the rotation angle (yaw) around the normal direction of the calibration board is typically 3-5 times that of other rotation angles. This lack of observability is particularly prominent in large-scale applications such as automotive systems. Secondly, existing calibration procedures suffer from significant engineering implementation bottlenecks. To obtain reliable calibration results, traditional methods require the calibration board to complete a complex "figure-eight" motion trajectory within the camera's field of view. According to ETH Zurich's 2020 experimental data, at least 50 sets of image data in different poses are required to achieve satisfactory calibration accuracy, with the entire process taking 30-45 minutes. This not only increases calibration costs but also presents practical difficulties due to limited operating space for large equipment such as automotive systems. Furthermore, controlling the speed of the calibration board's movement is crucial: too slow a movement leads to dominant IMU noise, while too fast a movement causes image motion blur; this contradiction is extremely difficult to manage in engineering practice. A deeper problem lies in the step-by-step calibration strategy of spatiotemporal parameters employed in existing methods. This sequential processing approach, prioritizing space over time, leads to the progressive propagation and accumulation of errors. Tests conducted by the Bosch autonomous driving team in 2021 showed that a 5ms deviation in time synchronization resulted in an 8cm positioning error at a vehicle speed of 60km / h. In dynamic calibration scenarios, such as online calibration of drones in hovering states, this error accumulation effect is further amplified. Furthermore, the sensitivity of 2D calibration boards to ambient lighting conditions (such as reflections and shadows) severely limits their reliability in industrial applications.

[0004] With the diversification of application scenarios and the continuous improvement of performance requirements, existing two-dimensional calibration board methods are increasingly unable to meet practical needs. In the field of autonomous driving, next-generation intelligent vehicles require sensor calibration to not only be highly accurate but also to be completed quickly within a limited space; in industrial robot applications, the calibration process needs to adapt to harsh environments such as vibration and oil contamination; for consumer-grade AR / VR devices, the calibration scheme must be simple, easy to use, and cost-effective. These demands all point to a common technological breakthrough direction: developing a new calibration scheme that can overcome the inherent defects of two-dimensional calibration boards. In recent years, some research has begun to explore the possibility of three-dimensional calibration structures. For example, the Trihedron three-sided calibration board developed by ETH Zurich improves parameter observability through orthogonal plane design; the stepped calibration board proposed by Stanford solves the scale ambiguity problem by utilizing known height differences. Although these attempts have shown some potential, they still have significant shortcomings in terms of calibration algorithm optimization and engineering practicality. In particular, how to achieve joint optimization of spatiotemporal parameters, how to adapt to modular design for different application scenarios, and how to improve the automation level of the calibration process all require more systematic solutions.

[0005] In summary, existing Camera-IMU joint calibration methods have significant shortcomings in terms of technical accuracy, engineering operability, and system adaptability. There is an urgent need for a new calibration scheme with breakthroughs in structural design, modeling methods, and optimization strategies to improve overall system performance, lower deployment threshold, and promote the large-scale application of multi-sensor fusion systems in automated equipment.

[0006] Therefore, existing technologies still need to be improved. Summary of the Invention

[0007] In view of the shortcomings of the existing technologies, especially the structural deficiencies in calibration observability, system robustness and parameter coupling processing, this invention proposes a Camera-IMU calibration method for unmanned automated equipment that integrates three-dimensional structure and nonlinear joint optimization, aiming to achieve a multi-sensor extrinsic parameter calibration process with higher accuracy, stronger adaptability and higher level of engineering automation.

[0008] The technical solution of the present invention is as follows:

[0009] This invention provides a camera-IMU joint calibration method for unmanned automated equipment, the method comprising the following steps:

[0010] S1. By acquiring image frames containing three-dimensional calibration structures, extract the corner coordinates of multiple planes and calculate the homography matrix corresponding to each plane;

[0011] S2. Based on the homography matrix, the singular value decomposition (SVD) method is used to recover the camera pose relative to each plane, and the initial pose of the camera in multiple frames is obtained.

[0012] S3. Utilize the corner reprojection error of multi-frame images to construct a bundle adjustment optimization problem and jointly optimize the camera pose of all frames.

[0013] S4. Perform time interpolation on the IMU or GPS trajectory to align it with the time of the image frame, thereby obtaining continuous IMU pose.

[0014] S5. Construct a hand-eye calibration model based on the relative motion between the camera and the IMU, solve the rotation parameters using quaternion modeling, and further estimate the translation parameters;

[0015] S6. Introduce the initial values ​​of rotation and translation into the joint optimization model, and optimize the extrinsic parameters between the camera and the IMU using the nonlinear least squares method.

[0016] In one embodiment, the three-dimensional calibration structure includes at least three non-parallel planes, each with a detectable corner pattern.

[0017] In one embodiment, the homography matrix is ​​calculated using a direct linear transformation between image coordinates and calibration board model points, combined with camera intrinsic parameters for distortion correction.

[0018] In one embodiment, the beam adjustment optimization employs a sparse matrix solver with a robust loss function to weight the matching of anomalous corner points.

[0019] In one embodiment, the hand-eye calibration model is modeled in the following form:

[0020] AX = XB:

[0021]

[0022] In the formula, {C k} and {I k} represents the camera pose in the k-th frame and the interpolated GPS / IMU pose at that time, respectively. For {I k} to {I k+1 The relative pose of}. From {C k} to {C k+1 The relative pose of}.

[0023] In one embodiment, the rotation parameters are expressed in quaternion form, which are then converted into an overdetermined linear system of equations by combining left and right multiplication matrices, and the optimal quaternion solution is obtained through SVD decomposition.

[0024] In one embodiment, when the vehicle motion is planar, a simplified model is constructed by constraining the Z-axis translation to zero, thereby improving the stability of the translation parameter estimation.

[0025] In one embodiment, the translation parameters are obtained by establishing a linear equation AX = b in block matrix form and solving it using the least squares method.

[0026] In one embodiment, the joint optimization is solved iteratively using the Levenberg-Marquardt algorithm, with camera reprojection error and IMU trajectory residuals as joint loss terms.

[0027] In one embodiment, the calibration method is applicable to sensor calibration tasks in autonomous driving systems, mobile robots, augmented reality devices, or industrial automation platforms.

[0028] The Camera-IMU joint calibration method proposed in this invention, based on a joint modeling mechanism of three-dimensional calibration structure and spatiotemporal parameters, has the following significant technical advantages and unexpected beneficial effects compared with existing two-dimensional calibration methods:

[0029] 1. Significantly improves space observability and avoids degradation issues.

[0030] Employing a three-dimensional multi-faceted calibration structure provides independent geometric constraints along each axis, significantly reducing Z-axis translation and Yaw angle estimation errors, and addressing the technical blind spot where two-dimensional planar structures cannot be observed from specific viewpoints. Actual tests show that the Z-axis translation error is reduced from ±15cm using the original method to within ±3cm, and the standard deviation of the Yaw angle is reduced by more than 40%.

[0031] 2. Introduce a multi-frame joint optimization mechanism to improve pose estimation accuracy.

[0032] By globally optimizing the corner reprojection error between multiple frames using bundle adjustment, a holistic fitting of the camera pose sequence can be achieved, reducing local drift. This mechanism effectively improves the stability of image matching, reducing the camera odometry drift rate to less than 0.3%, significantly outperforming traditional frame-by-frame estimation methods.

[0033] 3. A spatiotemporal joint modeling strategy is adopted to suppress the error accumulation effect.

[0034] By merging time synchronization and spatial extrinsic parameters into the same set of optimized variables, the problem of error propagation at each level in traditional serial calibration methods is avoided, ensuring the overall consistency and coupling stability of the system, and making it suitable for real-time calibration requirements in dynamic systems.

[0035] 4. Enhanced engineering adaptability, suitable for various automated equipment scenarios.

[0036] The calibration structure features high spatial visibility and flexible installation. The calibration process does not rely on complex trajectory movements, making it particularly suitable for applications with limited space or complex operating environments, such as autonomous vehicles, industrial robotic arms, and AR / VR terminals. Deployment efficiency is improved by more than 70%.

[0037] 5. Supports planar motion modeling, with strong adaptability and high computational efficiency.

[0038] In typical planar motion systems such as vehicle-mounted or wheeled robots, an optional "Z-axis constraint model" is integrated. By solving the translation terms through structured linear equations, the translation accuracy and model convergence speed are significantly improved, while reducing the consumption of computing resources.

[0039] 6. The optimized process is well-encapsulated and suitable for modular integration and automated invocation.

[0040] Each computing module is encapsulated with a standardized interface, supporting direct calls to the ROS framework or integration into the C++ / Python environment. It can be quickly deployed to existing sensing or positioning systems and has good system compatibility and engineering portability.

[0041] In summary, this invention significantly improves the accuracy, robustness, efficiency, and deployability of Camera-IMU joint calibration through structural innovation, modeling method upgrades, and optimization algorithm integration. It possesses outstanding engineering practical value and industrial application potential, and breaks through the inherent technical bottlenecks of traditional two-dimensional calibration methods. Attached Figure Description

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0043] Figure 1 A schematic diagram of a three-dimensional calibration board for the Camera-IMU joint calibration method for unmanned automated equipment provided by the present invention;

[0044] Figure 2 This is a flowchart illustrating the implementation of the Camera-IMU joint calibration method for unmanned automated equipment provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments of the invention are described below in conjunction with the accompanying drawings.

[0046] This invention provides a camera-IMU joint calibration method for unmanned automated equipment. Please refer to [link to relevant documentation]. Figures 1-2 The method includes the following steps:

[0047] S1. By acquiring image frames containing three-dimensional calibration structures, extract the corner coordinates of multiple planes and calculate the homography matrix corresponding to each plane;

[0048] S2. Based on the homography matrix, the singular value decomposition (SVD) method is used to recover the camera pose relative to each plane, and the initial pose of the camera in multiple frames is obtained.

[0049] S3. Utilize the corner reprojection error of multi-frame images to construct a bundle adjustment optimization problem and jointly optimize the camera pose of all frames.

[0050] S4. Perform time interpolation on the IMU or GPS trajectory to align it with the time of the image frame, thereby obtaining continuous IMU pose.

[0051] S5. Construct a hand-eye calibration model based on the relative motion between the camera and the IMU, solve the rotation parameters using quaternion modeling, and further estimate the translation parameters;

[0052] S6. Introduce the initial values ​​of rotation and translation into the joint optimization model, and optimize the extrinsic parameters between the camera and the IMU using the nonlinear least squares method.

[0053] Specifically, in this embodiment, a Camera-IMU joint calibration method for unmanned automated equipment includes multiple stages such as image acquisition, corner extraction, pose estimation, trajectory interpolation, hand-eye modeling, initial value solution and nonlinear optimization.

[0054] First, several image frames containing the 3D calibration structure are acquired using the camera on the target device, and the corresponding raw IMU data and timestamps are recorded. Image processing algorithms are then used to detect multiple checkerboard corner points on the 3D structure in the images, and image distortion is corrected. Subsequently, based on the extracted corner point information, homography matrices for multiple planes are constructed.

[0055] Based on this, singular value decomposition (SVD) is used to recover the relative pose between the camera and each plane, obtaining a preliminary camera trajectory. The corner reprojection error between all image frames is used as the optimization objective, and bundle adjustment is employed to jointly optimize the entire trajectory. Simultaneously, temporal interpolation is performed on the IMU / GPS trajectory to align it with the camera trajectory, constructing a hand-eye calibration model. Finally, nonlinear optimization methods are used to further refine the extrinsic parameters, achieving calibration closure.

[0056] In practice, the entire calibration process should ideally be completed in a static environment to ensure that there are no drastic changes in lighting or background interference during image acquisition. It is recommended to acquire at least 20 image frames, covering multiple viewpoints (including low angle, oblique angle, close range, and long range) to enhance the robustness of pose estimation.

[0057] The raw IMU data should include acceleration and angular velocity data along three axes. The sampling frequency is recommended to be no less than 200Hz, and the timestamp recording should be accurate to the microsecond level to ensure the quality of subsequent interpolation.

[0058] All raw data (image frames, IMU data) must be stored as timestamp-synchronized data packets and uniformly encoded for later processing and analysis.

[0059] In a further embodiment, the three-dimensional calibration structure includes at least three non-parallel planes, each plane having a detectable corner pattern.

[0060] The three-dimensional calibration structure consists of three non-parallel checkerboard surfaces, each with a high-contrast corner dot pattern (such as a classic checkerboard) with a fixed arrangement. The three surfaces are fixed within a unified structure by an integrated frame, maintaining a strict spatial angle (preferably approximately orthogonal).

[0061] To enhance visibility, all surfaces are coated with a matte finish to reduce glare, and the printing accuracy of corner patterns is controlled within ±0.1mm. The calibration plate features a lightweight design, facilitating movement and mounting, and is suitable for calibration deployment in various equipment spaces.

[0062] In a further embodiment, the homography matrix is ​​calculated using a direct linear transformation between image coordinates and calibration plate model points, combined with camera intrinsic parameters for distortion correction.

[0063] By extracting the pixel coordinates of each corner point in the 3D structural image, and utilizing the direct linear transformation relationship between the known checkerboard geometric model point coordinates and the image coordinates, a homography matrix corresponding to each plane is constructed.

[0064] In this embodiment, Zhang's calibration method is used as the base model, and the camera intrinsic parameters are combined with the diagonal coordinates for anti-distortion processing to ensure high robustness and accuracy in the homography matrix solution process. For each plane, its projection relationship is independently constructed to ensure sufficient constraints in the subsequent pose recovery process.

[0065] In a further embodiment, the beam adjustment optimization employs a sparse matrix solver with a robust loss function to perform weighted processing on abnormal corner point matching.

[0066] Specifically, after solving the initial camera pose, this implementation uses Ceres Solver to construct a joint optimization graph, takes the corner reprojection error between all image frames as the objective function, and introduces a robust loss function (such as Huber or Cauchy) to weight and control the influence of outliers on the optimization solution.

[0067] The parameter space is iteratively solved by gradient descent strategy, and the optimized variables include the spatial coordinates of the camera pose and 3D structure points in each frame. Finally, a smoother and more accurate camera trajectory is obtained, which provides a reliable input for subsequent extrinsic parameter estimation.

[0068] In a further embodiment, the hand-eye calibration model is modeled in the following form:

[0069] AX = XB:

[0070]

[0071] In the formula, {C k} and {I k} represents the camera pose in the k-th frame and the interpolated GPS / IMU pose at that time, respectively. For {I k} to {I k+1 The relative pose of}. From {C k} to {C k+1 The relative pose of}.

[0072] This model ensures the overall consistency of the rotation and translation relationships and naturally adapts to the time-aligned pose pair input, avoiding additional coupling introduced in parameter solving.

[0073] In a further embodiment, the rotation parameters are expressed in quaternion form, and are converted into an overdetermined linear equation system by combining left and right multiplication matrices, and the optimal quaternion solution is obtained by SVD decomposition.

[0074] The rotational portion of the hand-eye model is handled using quaternions. The original rotational relationships are transformed into quaternion multiplication forms, and then rewritten into linear matrix form using left-right multiplication by a quaternion transformation matrix.

[0075] Aq = 0,

[0076] Where q is the target quaternion and A is the coefficient matrix composed of rotation pairs.

[0077] The overdetermined linear system is solved using SVD, and the right singular vector corresponding to the minimum singular value is extracted as the optimal rotation solution. Regularization is used to ensure that q is a unit quaternion, thereby restoring a stable initial value for rotation.

[0078] In a further embodiment, when the vehicle motion is planar motion, a simplified model is constructed by constraining the Z-axis translation to zero, thereby improving the stability of the translation parameter estimation.

[0079] For devices with only planar motion (such as wheeled mobile robots), this invention introduces the assumption of Z-axis unobservability, simplifying translation modeling to a two-dimensional model under the premise of stable initial rotational values:

[0080] Δx = R·ΔX + t,

[0081] Where Δx is the translation difference on the image side, ΔX is the translation on the IMU side, R is the estimated rotation, and t is the translation term to be solved.

[0082] This simplified model avoids instability caused by Z-axis drift and improves the solution speed.

[0083] In a further embodiment, the translation parameters are obtained by establishing a linear equation AX = b in block matrix form and solving it using the least squares method.

[0084] The above planar motion constraint equations are rewritten in standard block matrix form AX = b, where X is the target translation vector.

[0085] In this embodiment, multiple relative pose pairs are superimposed to construct a unified large-scale sparse linear system, and the least squares solution is obtained by using QR decomposition or conjugate gradient method to obtain translation parameter estimation results with statistical optimality.

[0086] In a further embodiment, the joint optimization is solved iteratively using the Levenberg-Marquardt algorithm, with camera reprojection error and IMU trajectory residual as joint loss terms.

[0087] After obtaining the initial values ​​for rotation and translation, a joint optimization objective function is constructed, with the residual term including camera reprojection error and IMU trajectory error. The Levenberg-Marquardt algorithm is used to jointly optimize the extrinsic parameters, setting a maximum number of iterations and a convergence threshold to ensure stable convergence to a local optimum.

[0088] The damping factor is dynamically adjusted during the optimization process, and residual evaluation is performed in each round to control computational overhead. The optimization termination conditions include dual constraints of residual convergence rate and maximum number of iterations.

[0089] This method exhibits excellent platform adaptability, suitable for deployment in high-end autonomous driving systems as well as lightweight AR terminals or industrial robot systems. The optimization module employs a modular interface design, supporting integration with ROS, CUDA, and standard C++, thus meeting the needs for rapid calibration and batch deployment across multiple scenarios.

[0090] The miniaturized design of the calibration structure facilitates indoor and outdoor mobile operation, and the algorithm can run offline and supports edge computing deployment, making it feasible for implementation in practical engineering projects.

[0091] To make the implementation of the method of this invention clearer, the key processes are summarized as follows:

[0092] Camera pose calculation:

[0093] (1) Calculation of plane normal vector of three-dimensional chessboard calibration board

[0094] Suppose the 3D calibration board consists of three independent checkerboard planes ∏1, ∏2, and ∏3, with normal vectors n1, n2, and n3 for each plane, and that no two planes are parallel. The coordinates of each checkerboard corner point in its local coordinate system are... (i = 1, 2, 3).

[0095] (2) Homography matrix calculation

[0096] For each chessboard square Π i Image coordinates are obtained through corner detection. Calculate the homography matrix H i :

[0097]

[0098] Where π(·) is the projection function. Let K be the homogeneous coordinates and K be the camera intrinsic parameter matrix.

[0099] (3) Planar pose solution

[0100] The relative poses of the camera and the checkerboard are decomposed from the homography matrix.

[0101]

[0102] The rotation matrix is ​​obtained by orthogonalization:

[0103]

[0104] Construct the camera's pose to each chessboard grid:

[0105]

[0106] (4) Multiplane joint optimization

[0107] The pose constraints of the three chessboard grids are jointly optimized using bundle adjustment (BA), and the objective function is:

[0108]

[0109] in, The pose of the camera in the world coordinate system. The pose of each chessboard square in the world coordinate system.

[0110] Joint calibration:

[0111] The hand-eye calibration equation AX = XB is constructed from the relative pose constraints of the camera odometry and the interpolated GPS / IMU odometry obtained above.

[0112]

[0113] In the formula, {C k} and {I k} represents the camera pose in the k-th frame and the interpolated GPS / IMU pose at that time, respectively. For {I k} to {I k+1 The relative pose of}. From {C k} to {C k+1 The relative pose of}.

[0114] The hand-eye calibration equation established by the above formula can be decomposed into two parts: rotation terms and translation terms.

[0115]

[0116] Solving for the rotation term yields the translation term. However, a reliable initial value for the rotation is needed. The rotation term in the above equation is written in quaternion form, and then transformed into matrix multiplication using left and right quaternion multiplication matrices to obtain the following formula:

[0117]

[0118] in It is a quaternion multiplication operator, and and These are the matrix representations of left and right quaternion multiplication, respectively.

[0119] After accumulating measurement data at different times, we obtain the following overdetermined equation:

[0120]

[0121] Where K is the number of rotation pairs in the overdetermined equations. These are robust weights used to better handle outliers. The difference angle of the current rotation pair on the angle axis is calculated by the above formula and used as a parameter of the Huber loss; its derivative is the weight.

[0122]

[0123] Where ρ() represents the Huber loss, q ω It is the real part of the quaternion q, and ()* denotes taking the inverse of the quaternion.

[0124] In this embodiment of the invention, the overdetermined equation above is solved using SVD, and its closed-form solution is the right-hand unit singular vector corresponding to the smallest singular value. Simultaneously, to ensure sufficient rotational constraints, the second smallest singular value needs to be greater than a set threshold. With... The rapid increase in the number of values ​​allows for the elimination of minimum rotation constraints through a priority queue, thus obtaining reliable initial rotation values. At this point, the translation term is solved by accumulating the relative poses from different time periods:

[0125]

[0126] However, in general, vehicle motion is usually planar motion with three degrees of freedom: x, y, and yaw. Therefore, the z-axis is usually not observable. Furthermore, since the IMU's acceleration is coupled with gravity, it is related to rotation. Therefore, using the initial rotational value obtained from the IMU measurement to calculate the initial translational value is unreliable. When the calculated z-axis translation value has a large deviation, we can make a planar assumption and rewrite the equation as follows:

[0127]

[0128] The above equation is the planar motion constraint generated by the (k+1)th relative pose, where γ is the yaw angle and t x and t y These are translations along the x and y axes, respectively, R k+1 yes The 2x2 block matrix in the top left corner, and yes and The first two elements of the column vector.

[0129] The equation can be rewritten in the form of AX = b:

[0130]

[0131] in, [t1k+1] i This represents the i-th element of the column vector [t1k+1].

[0132] By superimposing the measurements from different times according to the above equation, we can obtain the final matrix equation AX = b, which can be solved using the least squares method.

[0133]

[0134] Calibration and optimization solution:

[0135] Based on the obtained camera odometry pose, GPS / IMU odometry pose, and initial extrinsic parameters, construct the optimization objective function:

[0136]

[0137] By optimizing the cost function using the Levenberg-Marquardt algorithm, the final accurate extrinsic parameters can be obtained when the iteration converges.

[0138] In summary, this invention proposes a joint camera-IMU calibration method for unmanned automated equipment, systematically solving the long-standing technical bottlenecks of existing two-dimensional calibration techniques, such as insufficient observability, error accumulation, difficulty in time alignment, and poor engineering adaptability. By introducing a three-dimensional checkerboard calibration board, a multi-frame joint optimization mechanism based on bundle adjustment, a rotation solution strategy under quaternion modeling, and a time synchronization design that integrates IMU interpolation trajectories, a high-precision, high-efficiency, and highly adaptable external parameter joint calibration process is achieved.

[0139] This method is based on mathematical modeling and aims for engineering application, balancing accuracy and practicality. It exhibits high transferability in real-world engineering environments across multiple platforms, scenarios, and tasks.

[0140] In summary, this invention proposes a calibration solution with significant technological advancements and industrial value by breaking through and optimizing traditional methods. It not only fills the gap in existing technologies for three-dimensional structure calibration and spatiotemporal joint modeling, but also provides a sustainable and expandable technological foundation for the future development of multi-sensor fusion systems.

[0141] The various embodiments and technical details described in this specification serve to fully support the claims and disclose the technology. The path search method is not limited to specific parameter configurations and system implementations. All equivalent changes and improvements made based on the technical concept of this invention fall within the protection scope of this invention.

Claims

1. A camera-IMU joint calibration method for unmanned automated equipment, characterized in that, The method includes the following steps: S1. By acquiring image frames containing three-dimensional calibration structures, extract the corner coordinates of multiple planes and calculate the homography matrix corresponding to each plane; S2. Based on the homography matrix, the singular value decomposition (SVD) method is used to recover the camera pose relative to each plane, and the initial pose of the camera in multiple frames is obtained. S3. Utilize the corner reprojection error of multi-frame images to construct a bundle adjustment optimization problem and jointly optimize the camera pose of all frames. S4. Perform time interpolation on the IMU or GPS trajectory to align it with the time of the image frame, thereby obtaining continuous IMU pose. S5. Construct a hand-eye calibration model based on the relative motion between the camera and the IMU, solve the rotation parameters using quaternion modeling, and further estimate the translation parameters; S6. Introduce the initial values ​​of rotation and translation into the joint optimization model, and optimize the extrinsic parameters between the camera and the IMU using the nonlinear least squares method.

2. The Camera-IMU joint calibration method for unmanned automated equipment according to claim 1, characterized in that, The three-dimensional calibration structure includes at least three non-parallel planes, each with a detectable corner pattern.

3. The Camera-IMU joint calibration method for unmanned automated equipment according to claim 2, characterized in that, The homography matrix is ​​calculated by a direct linear transformation between the image coordinates and the calibration board model points, combined with camera intrinsic parameters for distortion correction.

4. The Camera-IMU joint calibration method for unmanned automated equipment according to any one of claims 1-3, characterized in that, The beam adjustment optimization employs a sparse matrix solver with a robust loss function to perform weighted processing on abnormal corner point matching.

5. The Camera-IMU joint calibration method for unmanned automated equipment according to any one of claims 1-4, characterized in that, The hand-eye calibration model is modeled in the following form: AX = XB: In the formula, {C k } and {I k } represents the camera pose in the k-th frame and the interpolated GPS / IMU pose at that time, respectively. For {I k } to {I k+1 The relative pose of}. From {C k } to {C k+1 The relative pose of}.

6. The Camera-IMU joint calibration method for unmanned automated equipment according to claim 5, characterized in that, The rotation parameters are expressed in quaternion form, and are converted into an overdetermined linear equation system by combining left and right multiplication matrices. The optimal quaternion solution is then obtained through SVD decomposition.

7. The Camera-IMU joint calibration method for unmanned automated equipment according to any one of claims 1-6, characterized in that, When the vehicle's motion is planar, a simplified model is constructed by constraining the Z-axis translation to zero, thereby improving the stability of the translation parameter estimation.

8. The Camera-IMU joint calibration method for unmanned automated equipment according to claim 7, characterized in that, The translation parameters are obtained by establishing a linear equation AX = b in block matrix form and solving it using the least squares method.

9. The Camera-IMU joint calibration method for unmanned automated equipment according to any one of claims 1-8, characterized in that, The joint optimization is solved iteratively using the Levenberg-Marquardt algorithm, with camera reprojection error and IMU trajectory residual as the joint loss term.

10. The Camera-IMU joint calibration method for unmanned automated equipment according to any one of claims 1-9, characterized in that, The calibration method is applicable to sensor calibration tasks in autonomous driving systems, mobile robots, augmented reality devices, or industrial automation platforms.

Images