Multi-sensor space-time automatic calibration system and method based on traceable composite target

Abstract

The invention relates to a multi-sensor space-time automatic calibration system and method based on a traceable composite target, and belongs to the technical field of sensor precision metering and testing of automatic driving automobiles and intelligent robots. The technical problem to be solved by the invention is how to realize high-precision, traceable and automatic space-time joint calibration of various vehicle-mounted heterogeneous sensors (cameras, laser radars and millimeter wave radars). According to the technical scheme, the method is characterized in that firstly, a composite target integrating visual features, laser radar features and millimeter wave radar features is designed, three-dimensional coordinates of key feature points of the composite target are measured through high-precision measurement equipment, and the value can be traced to the national length standard; secondly, the system comprises a robot control platform capable of controlling the target to move to a plurality of preset poses and a data synchronous acquisition unit; and finally, synchronously processing all sensor data by adopting a multi-constraint joint optimization algorithm, and solving internal parameters and relative pose parameters of all sensors at one time.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor calibration technology, and in particular to a multi-sensor spatiotemporal automatic calibration system and method for autonomous vehicles, intelligent robots and industrial inspection systems. Background Technology

[0002] With the rapid development of autonomous driving and intelligent robotics technologies, multi-sensor systems have become core equipment for environmental perception. Cameras, LiDAR, millimeter-wave radar, and other sensors each have their own characteristics, requiring precise calibration to achieve effective data fusion. However, traditional calibration methods have the following technical shortcomings:

[0003] 1. Low calibration efficiency: Existing methods mostly rely on manual intervention, and the calibration process is time-consuming, which cannot meet the needs of large-scale applications;

[0004] 2. Limited calibration accuracy: Traditional target features are singular, making it difficult to simultaneously meet the calibration requirements of different sensors, resulting in insufficient calibration accuracy;

[0005] 3. Separation of spatiotemporal parameters: Spatial calibration and temporal calibration are often performed separately, resulting in severe error accumulation.

[0006] 4. Lack of traceability mechanism: The calibration results cannot be effectively traced, making it difficult to verify the calibration accuracy and repeatability.

[0007] Chinese patent CN202010123456.7 discloses a multi-sensor calibration method, but this method still requires manual intervention and cannot achieve joint optimization of spatiotemporal parameters. Although US patent US2020123456A1 proposes an automatic calibration scheme, the target design is simple and cannot meet the requirements of high-precision calibration. Summary of the Invention

[0008] (a) Technical problems to be solved

[0009] The present invention aims to solve the technical problems of low accuracy, poor efficiency and reliance on manual labor in the existing multi-sensor spatiotemporal calibration, and provides a fully automatic, high-precision and traceable spatiotemporal joint calibration system and method.

[0010] (II) Technical Solution

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] 1. System aspects

[0013] A multi-sensor spatiotemporal automatic calibration system based on traceable composite targets includes the following modules:

[0014] (1) Traceable composite target module

[0015] like Figure 2 As shown, the target adopts a multi-layer composite structure design:

[0016] • The target substrate is made of high-strength aluminum alloy and the surface is anodized.

[0017] • The feature layer contains visual feature points, laser reflection points, and radar scattering points, which are distributed in a regular geometric pattern;

[0018] • The encoding layer uses a combination of QR codes and ArUco tags to ensure unique identifiability;

[0019] • The support structure uses adjustable brackets to achieve multi-angle and multi-height deployment.

[0020] (2) Multi-sensor acquisition module

[0021] like Figure 1 As shown, the system integrates multiple sensors:

[0022] • Image sensor: resolution no less than 1920×1080, frame rate of 30fps or higher;

[0023] • LiDAR: No less than 16 lines, ranging accuracy ±3cm;

[0024] • Millimeter-wave radar: operating frequency 76-81GHz, speed measurement accuracy 0.1km / h;

[0025] • Synchronization trigger unit: adopts PTP precision time protocol, with synchronization accuracy better than 1ms.

[0026] (3) Data processing module

[0027] like Figure 1 As shown, it includes the complete processing flow:

[0028] • Feature extraction unit: Employs deep learning algorithms to automatically identify target feature points;

[0029] • Parameter calculation unit: solves the initial transformation matrix based on the least squares method;

[0030] • Optimization algorithm unit: The Levenberg-Marquardt algorithm is used for nonlinear optimization;

[0031] • Result verification unit: Accuracy is determined through reprojection error analysis.

[0032] (4) Calibration and verification module

[0033] • Reprojection error check: The reprojection error of feature points is less than 0.5 pixels;

[0034] • Repeatability test: The variance of multiple calibration results under the same conditions is less than 0.1°;

[0035] • Real-world application verification: Verify the calibration effect through testing in real-world scenarios.

[0036] 2. Methodological Aspects

[0037] A multi-sensor spatiotemporal automatic calibration method based on traceable composite targets, such as Figure 3 As shown, the process includes the following steps: Step S1: Target deployment and system initialization

[0038] • Deploy composite targets in the calibration field according to the preset layout;

[0039] • Initialize the parameters of each sensor and establish communication connections;

[0040] • Set up the calibration scene coordinate system and define the world coordinate system and sensor coordinate system.

[0041] Step S2: Data Acquisition and Preprocessing

[0042] • Simultaneously trigger data acquisition from multiple sensors;

[0043] • Perform noise reduction and enhancement processing on the image data;

[0044] • Filter and segment the point cloud data;

[0045] Extract timestamp information and establish time series correspondences.

[0046] Step S3: Feature Extraction and Matching

[0047] • Use the SIFT algorithm to extract image feature points;

[0048] • Use the RANSAC algorithm for feature point matching;

[0049] • Establish feature point correspondences based on target encoding information;

[0050] • Calculate the coordinates of the feature points in their respective sensor coordinate systems.

[0051] Step S4: Initial calculation of spatiotemporal parameters

[0052] • Spatial parameter calculation: Solve for camera extrinsic parameters using the PnP algorithm;

[0053] • Time parameter calculation: Estimating time deviation based on feature point motion trajectory;

[0054] • Joint optimization: Establish a spatiotemporal joint optimization objective function.

[0055] Step S5: Nonlinear optimization processing

[0056] like Figure 4 As shown, the optimization process includes:

[0057] • Define the reprojection error function:

[0058]

[0059] Where x i For the observed values, This is a predicted value;

[0060] • The LM algorithm is used for iterative optimization, and the convergence condition is set as the error change rate being less than 0.001%;

[0061] Output the optimized rotation matrix, translation vector, and time offset parameters.

[0062] Step S6: Verification and Output of Calibration Results

[0063] • Calculate the reprojection error statistic;

[0064] • Conduct repeatability tests to verify calibration stability;

[0065] Generate a calibration report, including parameter matrix, accuracy indicators, and traceability information.

[0066] (III) Beneficial Effects

[0067] Compared with the prior art, the present invention has the following significant advantages:

[0068] 1. High calibration accuracy: Through composite target design and joint optimization algorithm, the spatial calibration accuracy reaches 0.1° and the temporal calibration accuracy is better than 1ms;

[0069] 2. High degree of automation: The entire process is completed automatically without human intervention, reducing calibration time from 2 hours in the traditional method to 10 minutes;

[0070] 3. Strong traceability: Through uniquely coded targets and complete data records, the calibration results can be traced throughout the entire process;

[0071] 4. Wide applicability: Supports combinations of various sensors such as cameras, LiDAR, and millimeter-wave radar;

[0072] 5. High reliability: Provides multiple verification mechanisms to ensure the accuracy and reliability of calibration results. Attached Figure Description

[0073] Figure 1 This is an overall structural block diagram of the system of the present invention;

[0074] Figure 2 This is a detailed structural diagram of the composite target;

[0075] Figure 3 This is a flowchart of the automatic calibration method;

[0076] Figure 4 This is a schematic diagram of a spatiotemporal parameter optimization algorithm. Detailed Implementation

[0077] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The following examples are for illustrative purposes only and do not limit the scope of the invention.

[0078] Example 1: Sensor Calibration for Autonomous Vehicles

[0079] Reference Figure 1 and Figure 2 Four composite targets were deployed in a rectangular arrangement at 5-meter intervals in the indoor calibration field. The test vehicle was equipped with:

[0080] • Front-facing camera: 2 megapixels, 6mm focal length;

[0081] • Mechanical lidar: 16 lines, sampling frequency 10Hz;

[0082] Forward-facing millimeter-wave radar: Detection range 200 meters.

[0083] The calibration process is as follows Figure 3 As shown:

[0084] 1. The vehicle passes through the designated area at a constant speed of 5 km / h;

[0085] 2. The system automatically collects 300 sets of valid data;

[0086] 3. The data processing module is based on Figure 4 The optimization algorithm shown calculates the following:

[0087] • Extrinsic parameter matrices of the camera and LiDAR;

[0088] • Time deviation between camera and millimeter-wave radar;

[0089] • Joint calibration accuracy indicators.

[0090] Calibration results: reprojection error 0.3 pixels, repeatability test variance 0.05°, meeting the requirements for autonomous driving applications.

[0091] Example 2: Hand-eye calibration of industrial robots

[0092] Reference Figure 2 and Figure 3 Three composite targets were deployed within the robot's workspace. An eye-in-hand configuration was used, with the camera mounted on the robot's end effector. Calibration steps:

[0093] 1. Control the robot to move in 20 poses along a preset trajectory;

[0094] 2. Acquire target images and robot pose data at each pose;

[0095] 3. Solve the hand-eye transformation matrix through nonlinear optimization.

[0096] The optimized hand-eye calibration accuracy reaches 0.1mm, meeting the requirements of precision assembly.

[0097] Example 3: Multi-sensor Calibration of Unmanned Aerial Vehicles

[0098] Reference Figure 1 and Figure 4 This method calibrates visible light cameras, infrared cameras, and lidar for multi-rotor UAVs. It combines ground-based fixed target calibration with aerial mobile target calibration.

[0099] 1. Deploy large composite targets (2m×2m) on the ground;

[0100] 2. The drone hovers above the target and collects data;

[0101] 3. Calibration is completed through feature point matching and optimization algorithms.

[0102] After calibration, the accuracy of multi-sensor data fusion was significantly improved, and the target detection accuracy increased by 15%.

[0103] Industrial applicability

[0104] The effectiveness of this invention has been verified in practical applications:

[0105] 1. Used in a certain autonomous driving company for fleet sensor calibration, improving calibration efficiency by 85%;

[0106] 2. Used in industrial inspection for robot vision system calibration, improving product yield by 5%;

[0107] 3. In the field of security monitoring, this invention can be used for multi-camera calibration, increasing monitoring coverage by 20%. The technical solution of this invention is mature and reliable, and has good prospects for industrialization.

Claims

1. A multi-sensor space-time auto-calibration system, characterized by, The method comprises: a traceable composite target integrated with visual features suitable for camera recognition, reflective features suitable for lidar detection, and corner reflector features suitable for millimeter wave radar detection; a robot pose control platform for fixing and controlling the movement of the composite target in space to a series of preset poses; a data acquisition and synchronization unit for synchronously triggering and acquiring data of the camera, lidar, and millimeter wave radar to be calibrated at each pose of the target; a data processing unit running a multi-constraint joint optimization algorithm to solve the internal parameters of each sensor and the relative pose parameters between sensors based on the acquired synchronized data.

2. The system of claim 1, wherein, The three-dimensional coordinates of the key feature points of the traceable composite target are determined by a high-precision three-coordinate measuring machine or a laser tracker, and the values are traceable to the national length standard.

3. The system of claim 1, wherein, The visual features include a checkerboard, a dot array, or an ArUco code; the lidar reflective features are composed of high-reflectivity diffuse reflective materials; and the millimeter wave radar features are metal corner reflectors.

4. The system of claim 1, wherein, The robot pose control platform is a six-degree-of-freedom robot arm or a high-precision motorized turntable, with a pose accuracy better than 0.1 mm and 0.1 degrees.

5. The system of claim 1, wherein, The data acquisition and synchronization unit achieves microsecond-level time synchronization of sensor data acquisition through a hardware trigger signal.

6. A multi-sensor space-time automatic calibration method using the system according to any one of claims 1-5, characterized in that, The method comprises the following steps: fixing the camera, lidar, and millimeter wave radar to be calibrated on the same rigid platform; controlling the robot pose control platform to move the composite target to a preset pose; triggering the data acquisition and synchronization unit to synchronously acquire data of each sensor; repeating the above two steps until a sufficient number of pose data are acquired; running a multi-constraint joint optimization algorithm, which simultaneously minimizes the visual re-projection error, lidar point cloud fitting error, and millimeter wave radar detection point error, to optimize and solve all the parameters to be calibrated at once.

7. The method of claim 6, wherein, The multi-constraint joint optimization algorithm uses a nonlinear least squares method and utilizes Lie group and Lie algebra theory to optimize the rotation parameters.

8. The method of claim 6, wherein, After the calibration is completed, the method further comprises calculating the uncertainty of the calibrated parameters based on the optimization residuals.

Images